Fighting API Compatibility On Fluentd Using "Black Magic"
•
6 likes•7,355 views
The document discusses Fluentd's changes to its plugin API between versions 0.12 and 0.14. In 0.14, the API was overhauled to separate entry points from implementations and introduce a plugin base class to control data and control flow. A compatibility layer was added to allow most 0.12 plugins to work unmodified in 0.14 by handling calls to overridden methods. However, plugins that override certain methods like #emit may cause errors due to changes in how buffering works.
Report
Share
Fighting API Compatibility On Fluentd Using "Black Magic"
- 1. Fighting API Compatibility On Fluentd Using "Black Magic" Jul 2 2016 in YAPC::Asia Hachioji #yapc8oji Satoshi "Moris" Tagomori (@tagomoris)
- 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
- 6. Fluentd v0.14 API Update • Everything changed :) • Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::* • Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
- 7. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins
- 8. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin
- 9. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
- 10. Basic Weapons: Class and Mixin in Ruby
- 11. Class and Subclass in Ruby class A #bar class B #bar super B.new.bar
- 12. class A #bar class B #bar super B.new.bar module M #bar Introducing Methods by Mixin
- 13. class A #bar class B #bar super B.new.bar module M #bar Singleton Class of Ruby #bar B.new.singleton_class
- 14. class A #bar class B #bar super b=B.new b.singleton_class.include M2 b.bar module M #bar Adding Methods on An Instance (1) B.new.singleton_class #bar M2 #bar
- 15. class A #bar class B #bar super b=B.new b.extend M2 b.bar module M #bar Adding Methods on An Instance (2) B.new.singleton_class #bar M2 #bar
- 16. Back to Fluentd code :)
- 17. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
- 18. Fluentd v0.12 Fluent::Output class Fluent::Output #emit(tag, es, chain) MyOutput Engine calls plugin.emit(tag, es, chain) @buffer
- 19. Fluentd v0.12 Fluent::BufferedOutput (1) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format(tag,time,record) #format_stream(tag,es)
- 20. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
- 21. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class TimeSlicedOutput #format(tag,time,record)
- 22. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class ObjectBufferedOutput
- 23. Fluentd v0.12 Fluent::BufferedOutput class Fluent::Outputclass BufferedOutputMyOutput @buffer calls #write in OutputThread @buffer chunk #write(chunk) OutputThread #pop
- 24. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class TimeSlicedOutput OutputThread #write(chunk) @buffer calls #write in OutputThread #write calls chunk.key chunk #pop
- 25. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class ObjectBufferedOutput OutputThread #write(chunk) #write(chunk) #write_object(chunk_key, chunk) @buffer calls #write in OutputThread chunk #pop
- 26. Fluentd v0.12 API Problems • Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes • counting input events • hook arguments/return values to update API • Fluentd core didn't show fixed API • Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for arguments/return values
- 27. How can we solve this problem?
- 29. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
- 30. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput Output calls plugin.write (or try_write) @buffer chunk #write(chunk) #try_write(chunk) flush thread #process(tag, es) #format(tag, time, record)
- 31. Fluentd v0.14 Design Policy • Separate entry points from implementations • Methods in superclass control everything • Do NOT override these methods! • Methods in subclass do things only for themselves • not for data flow, control flow nor others • Plugins have simple/straightforward call stack • Easy to understand/maintain
- 32. ❤
- 33. How about existing v0.12 plugins?
- 34. Requirement: (Almost) All Existing Plugins SHOULD Work Well WITHOUT ANY MODIFICATION
- 35. • Fluent::Compat namespace for compatibility layer v0.14 Plugins & Compat Layer F::P::Output F::P::Base v0.14 Plugins Fluent:: Compat:: Output F::C:: Buffered Output F::C:: TimeSliced Output F::C:: ObjectBuffered Output Fluent::Output F:: Buffered Output F:: TimeSliced Output F:: ObjectBuffered Output v0.12 Plugins
- 36. Double Decker Compat Layer? • Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-( • Separate code into Fluent::Compat • and import it into Fluent top level
- 37. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
- 38. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
- 40. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (virtual) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 41. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (real) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 42. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 43. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread default implementation for calling "super"
- 44. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 45. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread When plugin overrides #emit
- 46. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread This call doesn't happen, in fact #emit doesn't return values! When plugin overrides #emit
- 47. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit calls @buffer.emit → NoMethodError !
- 48. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 49. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit 1. #emit calls @buffer.emit with data to be written in buffer 0. plugin calls @buffer.extend to add #emit 2. @buffer.emit stores arguments into plugin's attribute 3. get stored data 4. call @buffer.write with data
- 50. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
- 51. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Thinking about "chunk" instance ... Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #write may call "chunk.key", but v0.14 chunk doesn't have #key !
- 52. Fluent::Plugin::Outputclass MyOutput @buffer #write Compat::BufferedOutput #write(chunk) flush thread "chunk" has #metadata, and values of #key can be created via #metadata Let's "chunk.extend" ! Where to do so? ? Thinking about "chunk" instance ...
- 53. Fluent::Plugin::OutputMyOutput @buffer #write C::BufferedOutput #write(chunk) flush thread Thinking about "chunk" instance ... #write(chunk) BufferedChunkMixin plugin.extend BufferedChunkMixin in #configure
- 54. Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...
- 56. Plugin Lifecycle Updated Methods(v0.12) • #configure • #start • #before_shutdown • #shutdown v0.12 Plugins often doesn't call "super"! Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate In v0.14, these methods MUST call "super" • #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?
- 57. For Example: shutdown compat plugins Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown It doesn't call "super"! We want to call this...
- 58. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. Fluentd core calls #shutdown 2. call #shutdown? to check "super" is called or not 3. call #shutdown of superclass forcedly!
- 59. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown How to make this point?
- 61. class A #bar class B #bar super B.new.bar Wrapping Methods on a Class (1) B.new.singleton_class #bar
- 62. class A #bar class B #bar super B.new.bar module M Wrapping Methods on a Class (2) B.new.singleton_class #bar #bar Using extend is powerful, but it should be done for all instances How about wrapping methods for all instances of the class?
- 63. class A #bar class B #bar super module M;def bar;super;end;end B.prepend M B.new.bar module M Wrapping Methods on a Class (3): Module#prepend B.new.singleton_class #bar #bar module M wraps B, and M#bar is called at first
- 64. class A #bar class B #bar super b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar
- 65. class A #bar class B #bar super module M Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar Singleton class is a class, so it can be prepended :) b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend M b.bar #bar It's actually done in Test Driver implementation...
- 66. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown THIS ONE !!!
- 68. What We Got :-) Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. prepend CallSuperMixin at first 2. call #shutdown? to check "super" is called or not 3. if not, get method of superclass, bind self with it, then call it Thank you @unak -san!
- 70. IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P
- 71. Do Whatever You Can For Users! It Makes Everyone Happier ... Except for Maintainers :(