WebKit and GStreamer
- 2. WebKit and GStreamer Bringing lolcatz videos to your desktop Xabier Rodríguez Calvar Igalia
- 3. Who is this dude? · Born in the Basque Country · Grown and living in Galicia - One of the seven Celtic Nations, like Scotland · Studied Computer Science at Universidade da Coruña · Partner and developer at Igalia - Working in Multimedia and WebKit
- 5. Index · Quick review of WebKit · WebKit Multimedia · Power features - WebAudio - Hardware acceleration - Accelerated video compositing 5/36
- 6. Quick Review
- 7. WebKit · Web content engine · Processes and renders web content · Started as a fork of KTHML and KJS · Open source since 2005 · Goals: open source, compatibility, compliance, stability, performance, security, portability, usability, hackability · Non-goals: being a full web browser, being a science project, having reusable components, unlimited scope · Forked by Google as Blink · Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows 7/36
- 9. WebKitGTK+ · Each WebKit port is composed of - Platform interfaces - API layer · WebKitGTK+ platform layer: - libsoup for networking - cairo for rasterization - OpenGL for making the scene graph and WebGL - GStreamer for media - Various GTK+ APIs for talking with the system · API layer is a GtkWidget and a set of GObject APIs · WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp 9/36
- 10. WebKit2 · Give the web rendering parts of WebKit their own process · Page crashes don't crash the browser · Can put vulnerable data into a separate address space · Sandbox web rendering · Prevent pages from accessing the disk and operating system interface 10/36
- 14. WebKit Multimedia · Mac port: QuickTime, QtKit, AVFoundations · Apple Win port: QuickTime · GTK+ and EFL ports: GStreamer 1.0 backend · Blackberry port: Bridges over to its own Media playback framework · Chromium: bridges over to WebKit/Chromium backend which relays to FFMPeg implementations · Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on Windows 14/36
- 15. 15/36 remaertSG mroftalP - - - - - - - - - - - reyalPaideM eroCbeW tiKbeW WebKit Multimedia Architecture
- 16. GStreamer MediaPlayer - Overview · Mainly developed and tested on Linux platform · Successfully ported to MacOS/Win platfoms (not fully upstream yet though) · Playbin-based: the best GStreamer element for playback usecases - Playbin2 with GStreamer 0.10 · MediaPlayer registers and uses two WebKit-specific GStreamer elements: webkitwebsrc and a video-sink 16/36
- 17. GStreamer MediaPlayer - Main Features · On-disk-buffering · Frame-accurate seeking · Basic metrics reporting · Trick-modes · WebAudio · Hardware decoding support with VA-API (gst 1.2.x) · Video accelerated compositing · Codec installer support 17/36
- 18. WebKit and GStreamer Architecture MediaPlayerClient -eiPaeCin mdalyrlet +lyakhne( pabcCagd) +ouehne( vlmCagd) HTMLMediaElement -eiPae mdalyr +ly) pa( +lyakhne( pabcCagd) +eVlm(nvlm:la) stouei ouefot +ouehne( vlmCagd) -_rvt mpiae MediaPlayer +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateInterface +ly) pa( +eVlm(nvlm:la) stouei ouefot MediaPlayerPrivateGStreamerBase +eVlm(nvlm:la) stouei ouefot HTMLVideoElement HTMLAudioElement GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer +ly) pa( 18/36
- 19. GStreamer MediaPlayer - HTTP source element · a GStreamer bin implementing the URI handler interface for http(s):// and blob:// uris. · A single source pad provides raw data access to downstream elements. · appsrc is used internally to interface with the WebCore SubResourceLoader object · Element is registered with high rank in the GStreamer registry so playbin is more prone to pick it up to read http:// URIs. 19/36
- 20. GStreamer MediaPlayer - Video rendering · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input · Pass those to the MediaPlayerPrivateGStreamer via a signal 20/36
- 21. GStreamer MediaPlayer - Audio rendering · Use autoaudiosink · Check for pulseaudio for playback request messages · Pitch preservation · Preserve audio passthrough when possible · Volume managed by the sink 21/36
- 22. GStreamer MediaPlayer - Next steps · Improvements and bug fixes in the player: - Store on-disk buffered media in WebCore's cache directory - Make the webkitwebsrc element private to WebKit · MediaStream (getUserMedia) and WebRTC spec implementation · MediaSource adaptive streaming API spec implementation · Integration of WebAudio with media elements and mediastream stack · Finish porting to 1.2 · Remove GStreamer 0.10 code path 22/36
- 23. WebKit Multimedia Power Features
- 24. WebAudio · A JavaScript API allowing to create a new generation of Web games and applications · Effects, convolution engine for room simulation, etc · Realtime analysis, playback and visualization · W3C spec driven and implemented mainly by Chromium and Firefox · In WebKit, currently implemented for: - Mac - Chromium - GStreamer based Ports: GTK+, EFL, Qt 24/36
- 25. WebAudio GStreamer - Reading Audio · Spec allows two ways to read audio data: - from memory: we can use giostreamsrc GStreamer element - from a file path: we can use the filesrc GStremer element · Pipeline: src ! decodebin2 ! audioconvert ! audioresample ! capsfilter ! deinterleave ! queue ! appsink · deinterleave separates multi-channel audio to n audio mono channels (Pipeline above can contain multiple queue ! appsink chains) · appsink used to collect mono audio buffers, store them in a list and convert data to floating point values · Data is stored in a WebCore::AudioBus, containing one or multiple channel Float arrays 25/36
- 26. WebAudio GStreamer - Playing Audio · The WebAudio stack processed or generated some audio data to be sent to the sound card · Similarly to the "Reading Data" step, a WebCore::AudioBus is used to store the data · Custom GStreamer element to convert an AudioBus to an interleaved audio data flow: webkitwebaudiosrc · webkitwebaudiosrc: a bin with n internal pads pushing to an interleave element. Source pad outputs a single, interleave, wavencoded data flow · Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert ! autoaudiosink 26/36
- 27. WebAudio GStreamer - Work in Progress · Some layout tests don't pass yet · Not yet enabled by default in release builds · Support for stereo+ streams in the reader side. Playback side already handles those. · Integration with <video> and <audio> elements: WebAudio MediaElementSourceNode 27/36
- 28. Hardware Accelerated Video · We use GStreamer VA-API. (1.2.x) · That's it! Hardcore comes when compositing that. 28/36
- 29. Accelerated Video Compositing - Acceleration in WebKit From Joone Hur 29/36
- 30. Accelerated Video Compositing · Element inheriting from GstVideoSink - In practical sense, it is a GstAppSink with video capabilities negotiation · Takes video GstBuffers as input. · Video frame is transferred to the MediaPlayerPrivateGStreamer via a GLib signal. · MediaPlayerPrivateGStreamer uploads it to a GL texture and renders it asynchronously in the WebKit context. 30/36
- 31. Accelerated Video Compositing - Architecture TextureMapperPlatformLayer MediaPlayerPrivateInterface +an( pit) MediaPlayerPrivateGStreamerBase +an( pit) GstPlaybin -lyi pabn MediaPlayerPrivateGStreamer * GstElement ... WebKitVideoSink <cets> <rae> 31/36
- 32. Accelerated Video Compositing - Rendering paths · Software rendering path - It uses Cairo · Accelerated compositing paths - TextureMapperLayer: - Light-weight implementation of a scene-graph - The TextureMapperLayer tree is equivalent to the GraphicsLayer tree. - TextureMapperGL - Provides a GPU accelerated implementation of the drawing primitives, using shaders compatible with GL/ES 2.0 32/36
- 33. Accelerated Video Compositing - Next steps · More video decoders with GstVideoGLTextureUploadMeta support · Support EGLImage in WebKit as video frames handlers - And video decoders that deliver this kind of capabilities. · Handle the vertical blank interval in WebKit - GTK+ 3.8 seems to fix this issue. 33/36
- 36. Thank You! (q&a) twitter @calvaris www blogs.igalia.com/xrcalvar