[153] apache reef
•
27 likes•12,176 views
REEF is a meta-framework for big data analytics that eases development atop resource managers like YARN and Mesos. It provides a reusable control plane for coordinating data processing tasks and an adaptation layer for different resource managers. REEF decouples applications from cluster resources and handles common control plane functions like fault tolerance and configuration management. The framework is implemented in Java and C# and supports local, YARN, Mesos, and HDInsight execution environments. Future work includes graduating REEF from the Apache Incubator and using it to build new data processing frameworks and systems.
Report
Share
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to [153] apache reef
Similar to [153] apache reef (11)
More from NAVER D2
More from NAVER D2 (20)
Recently uploaded
Recently uploaded (20)
[153] apache reef
- 1. 딥러닝에서 스트림처리까지: 빅데이터 분석 메타 프레임워크 Apache(REEF 2015년 9월 14일 전병곤 서울대학교 컴퓨터공학부
- 2. REEF • A(meta-framework(that(eases(the(development(of(Big(Data( applications(atop(resource(managers(such(as(YARN(and(Mesos • Apache(Incubator(project(since(August(12,(2014 • 24(committers(– committers(from(SNU,(Microsoft,(Alibaba,( Purestorage,(UCLA,(UC(Berkeley,(UW,(SKT • Release(0.12.0-incubating(on(August(15,(2015
- 3. SNU(CMSLab Collaborators • Brian(Cho • Beom-Yeol Jeon • Joo-Seong Jeong • Dong-Jun(Lee • Yun-SeongLee • Young-Seok Yang • Gyeong-In(Yu • Young-Bin(Bok • Geon-Woo(Kim • Tae-Hun(Kim • Gye-Won(Lee • Woo-Yeon Lee • Tae-Geon(Um
- 4. Roadmap • What(is(REEF? • Why(would(you(want(REEF? • From(REEF(to(CAY
- 5. Current(Big(Data(Stack Interactive Query Business Intelligence Stream Processing Machine Learning Batch Processing Distributed File System Resource Manager (YARN, Mesos, etc.)
- 8. Common(Patterns • A(centralized(per-job(scheduler(that(observes(the(runtime(state(and( assigns(tasks(to(resources • A(runtime(for(executing(compute(tasks(and(retaining(state(in(an( organized(fashion • Communication(channels(for(monitoring(status(and(sending(control( messages • Configuration(management(for(passing(parameters(and(binding( application(interfaces(to(runtime(implementation
- 9. REEF Interactive Query Business Intelligence Stream Processing Machine Learning Batch Processing ! Reusable(control(plane(for(coordinating(data(plane(tasks ! Adaptation(layer(for(resource(managers ! Container(and(state(reuse(across(tasks(from(heterogeneous(frameworks ! Simple(and(safe(configuration(management ! Scalable(local,(remote(event(handling ! Java(and(C#((.NET)(support Distributed File System Resource Manager Services, Data Processing Lib REEF
- 10. Architectural(Choices(Summary 1. Scalable(master-workers(control(plane 2. Event(processing(framework:(Wake 3. Configuration(using(dependency(injection:(Tang 4. Container(and(state(reuse(across(heterogeneous(frameworks
- 14. REEF(Application(Component Driver Application(code(that(implements(the( resource(allocation(and(task(scheduling( components. 1. Runtime(Events 2. Evaluator(Events 3. Task(Events Task Application(code(executed(in(an( Evaluator.( Task(has(access(to(its(configuration(parameters(and(the Evaluator(state,(which(is(exposed(as(Contexts
- 17. REEF(Control(Flow:(Job(Life(Cycle Client public class DistributedShell { ... public static void main(String[] args)3{ ... Configuration4driverConf = DriverConfiguration.CONF. ... build(); ... reef.submit(driverConf); } } Resource(Manager Name(Node Data(Node HDFS Node(Manager YARN
- 18. Client public class DistributedShell { ... public static void main(String[] args)3{ ... Configuration4driverConf = DriverConfiguration.CONF. ... build(); ... reef.submit(driverConf); } } REEF(Control(Flow
- 19. public class DistributedShellJobDriver { private final EvaluatorRequestor requestor; ... public void onNext(StartTime time) { requestor.submit( EvaluatorRequest.Builder() .setSize(SMALL).setNumber(2) .build() ); } ... } REEF(Control(Flow Client
- 20. public class DistributedShellJobDriver { private final EvaluatorRequestor requestor; ... public void onNext(AllocatedEvaluator eval) { Configuration contextConf = ...; eval.submitContext(contextConf) } ... } REEF(Control(Flow Client
- 23. public class DistributedShellJobDriver { private final String cmd = “dir”; [...] public void onNext(ActiveContext ctx) { final String taskId = [...]; Configuration taskConf = Task.CONF .set(IDENTIFIER, "ShellTask") .set(TASK, ShellTask.class) .set(COMMAND, this.cmd) .build(); ctx.submitTask(taskConf); } [...] } task config REEF(Control(Flow Client
- 24. class ShellTask implements Task4{ private final String command; @Inject ShellTask(@Parameter(Command.class) String c) { this.command = c; } private String exec(final String command){ ... } @Override public byte[] call(byte[] memento) { String s = exec(this.cmd); return s.getBytes(); } } REEF(Control(Flow Client
- 30. Supported(Execution(Environments • Local(Machine:(testing • YARN:(Hadoop(resource(manager((containers) • Mesos:(resource(manager(started(at(UC(Berkeley((resource(offers) • HDInsight:(Azure(Hadoop(offering
- 31. Wake • Push-based(event(handlers,(observers • User(code:(Wake(event(handlers(that(react(to(various(REEF(events • Stage(to(execute(event(handlers:(synchronous(or(asynchronous( execution • Thread(management • Scalable(remote(messaging
- 32. Tang • Configuration(using(dependency(injection(that(restricts(dynamic(bind( patterns • Tang(specification:(binding(patterns • Bind(an(implementation(to(an(interface • Bind(a(value(to(a(configuration(parameter( • Tang(configuration(language(semantics( • “Configuration”(are(just(data:(cross-language(support • Configuration(options(can(be(set(at(most(once:(Immutability • A(large(subset(of(Tang’s(public(API(is(commutative
- 33. REEF(Implementation C# Java CPP Total Tang Wake REEF Services 8,951 4,677 13,821 4,527 7,454 5,268 24,281 11,620 0 0 1,957 0 16,405 9,945 40,059 16,147 Total 31,976 48,623 1,957 82,556 Lines(of(code(by(component(and(language((September(11,(2015) Applications(can(mix(and(match(Driver(side(event(handlers( in(Java(and(C#(with(any(number(of(Java(and(C#(Evaluators A(distributed(control(flow(framework(that(provides integration(across(language(boundaries
- 35. REEF(Project(Pointers • REEF(web(site:(http://reef.incubator.apache.org • Github mirrored(site:(https://github.com/apache/incubator-reef • Some(basic(examples(to(start(with • HelloREEF*:(HelloREEF,(HelloREEFYarn,(HelloREEFMesos,(… • TaskScheduler:(job(scheduler(to(schedule(embarrassingly(parallel(jobs(and( scale(out(and(in(resources(to(handle(fluctuating(demands • Tutorials:(https://cwiki.apache.org/confluence/display/REEF/Tutorials
- 39. Azure(Stream(Analytics(on(REEF • REEF(Driver • Compiles(and(optimizes((taking(user(budget(into(consideration)(a(query(into(a(data- flow(of(processing(stages • Uses(the(stage(data-flow(to(formulate(a(request(for(resources:(an(Evaluator(per(stage( instance • Driver(restartability • REEF(Task • Executes(the(stage(instance(work(on(an(assigned(partition • REEF(Services(used • Communication(service(built(on(Wake(that(allows(sending(messages(with(logical( identifiers • Checkpointingservice
- 40. VMWare(CORFU(on(REEF • CORFU((NSDI(2012):(a(distributed(logging(tool(providing(applications( with(consistency(and(transactional(services(at(extremely(high( throughput • CORFU(master(deployed(in(a(Driver • A(logging(unit(fails,(the(Driver(is(informed,(and(the(CORFU(master(reacts • REEF(Service(for(triggering(checkpointing and(restarting(a(Driver(from( a(checkpoint( • REEF(decouples(CORFU’s(resource(deployment(from(its(state,(allowing( CORFU(to(be(elastic
- 41. Why(Would(You(Want(REEF? • New(data(processing(framework • Cloud(Data(Runtime((CDR) • Iterative(Map-Reduce-Update((IMRU) • Elastic(ML(in(CAY • New(system(services( • Elastic(“memory”,(“communication”,(and(“cache”(in(CAY • Vortex:(harnessing(transient(cluster(resources(in(CAY
- 42. How(Does(REEF(Make(Life(Easier? • REEF(provides(high-level(component(abstraction:( Driver,(Task,(and(Service • REEF(application(developers(simply(write(handlers(that(react(to( various(events • REEF(runtime(handles(common(data(processing(control-plane( functionalities
- 43. What’s(Next? • We’re(ready(to(graduate(from(the(Incubator • Expected(to(be(the(3rd Apache(TLP(project(initiated(from(Korea( (after(Hama,(Tajo) • More(production(adoption • New(REEF(primitives • More(systems(built(on(top(of(REEF
- 44. From(REEF(To(CAY
- 45. CAY:(A(Unified(Big(Data(Analytics(Stack • 미래창조과학부 SW스타랩 프로젝트:( (SW(스타랩)(빅데이터(플랫폼(연구실 • Unification:(development,(operation • Big(Data(operating(system(layer • Unified(data(management • Unified(programming • New/enhanced(data(processing(engines
- 47. Elastic(Communication • Elastically(configure(communication(among(nodes • Elastic(shuffle • Elastic(group(communication • Scatter,(gather,(broadcast,(reduce,(…
- 48. Elastic(Machine(Learning((SNU,(Microsoft) • Goal:(optimize(large-scale(ML(execution(automatically • Built(on(“elastic”(system(services • Elastic(ML(runtime(&(optimizer • A(new(breed(of(elastic(Machine(Learning(algorithms(
- 51. Stream(processing • 초대용량 마이크로 스트림 질의 처리 플랫폼 (삼성미래기술육성사업) • Applications • Personalized(personal(digital(assistant • Real-time(disease(control • Smart(home,(smart(building
- 52. Vortex((SNU,(Microsoft(Research) • Harness(volatile(resources(available(in(the(datacenter • Large(idle(resources(in(production(systems • Massively(parallel(tasklets • Rearchitect distributed(analytics(execution((e.g.,(data(flow(graph( execution)(to(perform(well(in(such(volatile(environments
- 53. References • Web(links • http://reef.incubator.apache.org((REEF(project(site) • http://cmslab.snu.ac.kr (My(research(lab(site) • Publication • REEF:(Retainable(Evaluator(Execution(Framework,(ACM(SIGMOD(2015. • Making(Sense(of(Performance(in(Data(Analytics(Frameworks, USENIX(NSDI(2015 • Elastic(Memory:(Bring(Elasticity(Back(To(In-Memory(Big(Data(Analytics,(HotOS 2015.
- 55. 딥러닝에서 스트림처리까지: 빅데이터 분석 메타 프레임워크 Apache(REEF 2015년 9월 14일 전병곤 서울대학교 컴퓨터공학부 bgchun@snu.ac.kr