How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015

Google Cloud Dataflow is a fully managed service that allows users to build batch or streaming parallel data processing pipelines. It provides a unified programming model for batch and streaming workflows. Cloud Dataflow handles resource management and optimization to efficiently execute data processing jobs on Google Cloud Platform.

In my talk I will discuss and show examples of using Apache Hadoop, Apache Hive, Apache MXNet, Apache OpenNLP, Apache NiFi and Apache Spark for deep learning applications. This is the follow up to last years Apache Deep Learning 101 that was done at Dataworks Summit and ApacheCon. As part of my talk I will walk through using Apache NXNet Pre-Built Models, MXNet's New Model Server with Apache NiFi, executing MXNet with Apache NiFi and running Apache MXNet on edge nodes utilizing Python and Apache MiniFi. This talk is geared towards Data Engineers interested in the basics of Deep Learning with open source Apache tools in a Big Data environment. I will walk through source code examples available in github and run the code live on an Apache Hadoop / YARN / Apache Spark cluster. This will be an introduction to executing Deep Learning Pipelines in an Apache Big Data environment. My talk at Data Works Summit Sydney was listed in top 7 -> https://hortonworks.com/blog/7-sessions-dataworks-summit-sydney-see/ Also have speak at and run Future of Data Princeton and at Oracle Code NYC. https://www.slideshare.net/oom65/hadoop-security-architecture?next_slideshow=1 https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-apac.html https://dzone.com/refcardz/introduction-to-tensorflow

Apache Spark is a fast, general-purpose, and easy-to-use cluster computing system for large-scale data processing. It provides APIs in Scala, Java, Python, and R. Spark is versatile and can run on YARN/HDFS, standalone, or Mesos. It leverages in-memory computing to be faster than Hadoop MapReduce. Resilient Distributed Datasets (RDDs) are Spark's abstraction for distributed data. RDDs support transformations like map and filter, which are lazily evaluated, and actions like count and collect, which trigger computation. Caching RDDs in memory improves performance of subsequent jobs on the same data.

Apache Hadoop YARN is the modern distributed operating system for big data applications. In Apache Hadoop 3.1.0, YARN added a service framework that supports long-running services. This new capability goes hand in hand with the recent improvements in YARN to support Docker containers. Together these features have made it significantly easier to bring new applications and services to YARN. In this talk you will learn about YARN service framework, its new containerization capabilities and how it lays the foundation for a hybrid and uniform architecture for compute and storage across on-prem and multi-cloud environments. This will include examples highlighting how easy it is to bring applications to the YARN service framework as well as how to containerize applications. Here's what to expect in this talk: - Motivation for YARN service framework and containerization - YARN service framework overview - YARN service examples - Containerization overview - Containerization for Big Data and non Big Data workloads - wait that's everything

The document discusses Paytm Labs' transition from batch data ingestion to real-time data ingestion using Apache Kafka and Confluent. It outlines their current batch-driven pipeline and some of its limitations. Their new approach, called DFAI (Direct-From-App-Ingest), will have applications directly write data to Kafka using provided SDKs. This data will then be streamed and aggregated in real-time using their Fabrica framework to generate views for different use cases. The benefits of real-time ingestion include having fresher data available and a more flexible schema.

Telecom operators need to find operational anomalies in their networks very quickly. This need, however, is shared with many other industries as well so there are lessons for all of us here. Spark plus a streaming architecture can solve these problems very nicely. I will present both a practical architecture as well as design patterns and some detailed algorithms for detecting anomalies in event streams. These algorithms are simple but quite general and can be applied across a wide variety of situations.

Scalding is a scala DSL for Cascading. Run on Hadoop, it’s a concise, functional, and very efficient way to build big data applications. One significant benefit of Scalding is that it allows easy porting of Scalding apps from MapReduce to newer, faster execution fabrics. In this webinar, Cyrille Chépélov, of Transparency Rights Management, will share how his organization boosted the performance of their Scalding apps by over 50% by moving away from MapReduce to Cascading 3.0 on Apache Tez. Dhruv Kumar, Hortonworks Partner Solution Engineer, will then explain how you can interact with data on HDP using Scala and leverage Scala as a programming language to develop Big Data applications.​

Hadoop is becoming a standard platform for building critical financial applications such as risk reporting, trading and fraud detection. These applications require high level of SLAs (service-level agreement) in terms of RPO (Recovery Point Objective) and RTO (Recovery Time Objective). To achieve these SLAs, organizations need to build a disaster recovery plan that cover several layers ranging from the infrastructure to the clients going through the platform and the applications. In this talk, we will present the different architecture blueprints for disaster recovery as well as their corresponding SLA objectives. Then, we will focus on the stretch cluster solution that Crédit Agricole CIB is using in production. We will discuss the solution’s advantages, drawbacks and the impact of this approach on the global architecture. Finally, we will explain in detail how to configure and deploy this solution and how to integrate each layer (storage layer, processing layer...) into the architecture.

Developers increasingly are building dynamic, interactive real-time applications on fast streaming data to extract maximum value from data in the moment. To do so requires a data pipeline, the ability to make transactional decisions against state, and an export functionality that pushes data at high speeds to long-term Hadoop analytics stores like Hortonworks Data Platform (HDP). This enables data to arrive in your analytic store sooner, and allows these analytics to be leveraged with radically lower latency. But successfully writing fast data applications that manage, process, and export streams of data generated from mobile, smart devices, sensors and social interactions is a big challenge. Join Hortonworks and VoltDB, an in-memory scale-out relational database that simplifies fast data application development, to learn how you can ingest large volumes of fast-moving, streaming data and process it in real time. We will also cover how developing fast data applications is simplified, faster - and delivers more value when built on a fast in-memory, scale-out SQL database.

The document discusses a presentation on OpenStack Sahara given at a conference in Rome. It begins with introducing the three speakers and their backgrounds. It then provides an agenda for the presentation which includes an introduction to big data, an overview of OpenStack components, and a demonstration of Sahara in action. The presentation discusses what big data is, provides a brief history of MapReduce and Hadoop, and explains how OpenStack is well-suited to host big data platforms through its various components and architecture. It concludes by introducing OpenStack Sahara as a way to simplify deploying and managing Hadoop clusters on OpenStack.

NoSQL databases are being used everywhere by startups and Global 2000 companies alike for data environments that require cost-effective scaling. These environments also typically need to represent data in a more flexible way than is practical with relational databases.

DeepLearning4J (DL4J) is a powerful Open Source distributed framework that brings Deep Learning to the JVM (it can serve as a DIY tool for Java, Scala, Clojure and Kotlin programmers). It can be used on distributed GPUs and CPUs. It is integrated with Hadoop and Apache Spark. ND4J is a Open Source, distributed and GPU-enabled library that brings the intuitive scientific computing tools of the Python community to the JVM. Training neural network models using DL4J, ND4J and Spark is a powerful combination, but the overall cluster configuration can present some unespected issues that can compromise performances and nullify the benefits of well written code and good model design. In this talk I will walk through some of those problems and will present some best practices to prevent them. The presented use cases will refer to DL4J and ND4J on different Spark deployment modes (standalone, YARN, Kubernetes). The reference programming language for any code example would be Scala, but no preliminary Scala knowledge is mandatory in order to better understanding the presented topics.

Prezentace ze semináře ODA Partner Enablement Zámek Berchtold, 12.4.2018 Prezentoval Patrik Plachý,Senior Consultant Oracle

* Quick Intro to Bigtop * Trend Micro Big Data Platform * Mission-specific Platform * Big Data Landscape (3p) * Bigtop 1.1 Release (6p)

Sahara is an OpenStack service that allows users to easily provision and manage Hadoop clusters in OpenStack. It currently supports plugins for Hortonworks, Cloudera, and MapR distributions of Hadoop. The Cloudera plugin integrates Cloudera Manager to provision CDH services. Sahara aims to provide analytics as a service and allow data processing directly in OpenStack clusters using technologies like HDFS, Swift, and Hadoop frameworks. Performance overhead compared to bare metal Hadoop clusters is a current limitation being addressed.

The document discusses Cisco's Hadoop as a service offering on their Intercloud platform. Some key points: - Cisco provides managed Hadoop, including Cloudera's distribution, on optimized instances with local storage and object storage. This offers a scalable, reliable, and secure environment for Hadoop workloads. - Use cases discussed include predictive maintenance using IoT data and analyzing customer journeys across multiple channels. - A pilot test showed Cisco's platform could process over 100 million records from production data across various Hadoop jobs. - Cisco also discusses their data virtualization product CiscoDV, which can integrate data across on-premises, cloud sources on Cisco and AWS. -

Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop that was created by eBay and later open sourced as an Apache Incubator project. It provides security for Hadoop systems by instantly identifying access to sensitive data, recognizing attacks/malicious activity, and blocking access in real time through complex policy definitions and stream processing. Eagle was designed to handle the huge volume of metrics and logs generated by large-scale Hadoop deployments through its distributed architecture and use of technologies like Apache Storm and Kafka.

This document provides an overview of exploiting insecure IoT firmware. It begins with an introduction to IoT protocols like CoAP, MQTT, XMPP, and AMQP. It then discusses the OWASP top 10 security risks for IoT, focusing on insecure software/firmware. Common debugging interfaces for firmware like UART, JTAG, SPI, and I2C are explained. Operating systems and compilers used for IoT development are listed. Finally, the document outlines a methodology for exploiting insecure firmware, including getting the firmware, performing reconnaissance, unpacking, localizing points of interest, and then decompiling, compiling, tweaking, fuzzing, or pentesting the firmware. Tools mentioned include binwalk, firmwalk

NUMALLIANCE has successfully integrated new companies over the past 10 years to expand its capabilities in cold forming solutions for wire and tube. It recently merged with its competitor SILFAX, bringing increased tube expertise. While the companies previously competed in automotive and aerospace, they take different approaches that are now complementary within NUMALLIANCE. For example, SILFAX focuses on longer tube production while NUMALLIANCE develops solutions for other parts. The key to a successful merger is identifying the right competitors to combine capabilities while allowing team members to enhance operations rather than causing redundancy.

This document discusses various tools for data visualization, including D3.js, WebGL, the ELK stack, R, Processing, Open Refine, and 3D printing. It provides examples of visualizations created with each tool and suggests when each tool may be best to use. D3.js is described as a low-level library that provides full control but requires more work, while tools like the ELK stack allow for quickly visualizing system and business data. R is presented as useful for exploring and analyzing large datasets, and Open Refine is recommended for cleaning and preparing CSV files for export.

This document provides an overview of heterogeneous persistence and different database management systems (DBMS). It discusses why a single DBMS is often not sufficient and describes different types of DBMS including relational databases, key-value stores, and columnar databases. For each type, it outlines good and bad use cases, examples, considerations, and pros and cons. The document aims to help readers understand the different flavors of DBMS and how to choose the right ones for their specific data and access needs.

A report issued March 25, 2013 by the U.S. Geological Survey titled "Landscape Consequences of Natural Gas Extraction in Allegheny and Susquehanna Counties, Pennsylvania, 2004–2010." The report, using a series of maps and data, purports to show that drilling has lead to "carving up" wildlife habitats in some forests.

Suraj Pratap discusses security automation for red and blue teams. He outlines how he automates the server and application lifecycles using open source tools to address challenges around human capacity, tool selection, time, and cost when managing 600+ servers and 10+ applications across cloud infrastructures. Some areas he has automated include infrastructure security using Ansible and CloudFormation, security auditing using Scout2 and Prowler, offensive security tests using OpenVAS and Jenkins, vulnerability management with Dradis and Vulnreport.io, and security information and event monitoring with Alienvault and ELK.

Many vendors sell “security analytics” tools. Also, some organizations built their own security analytics toolsets and capabilities using Big Data technologies and approaches. How do you find the right approach for your organization and benefit from this analytics boom? How to start your security analytics project and how to mature the capabilities? (Source: RSA USA 2016-San Francisco)

Introduction to Java Management Extension (JMX) Technology

This session will share large scale architectures from the author's experiences with various companies like Cisco, Symantec, and EMC and compare and contrast the architecture across : Infrastructure Architecture Scaling, Ecommerce integrations and migration approach from legacy into AEM, Digital Marketing Cloud Integrations such as personalization, analytics, and DMP.

The document outlines a pilot leadership program called the Accelerated Leadership Class (ALC) at Peterson Air Force Base. The 7-session program will provide interactive leadership training to 12 junior airmen using experiential learning activities and models. Sessions will focus on developing leadership skills, emotional intelligence, giving and receiving feedback, and completing a group leadership project to benefit the base. The tentative schedule explores different timing options over 6 days, 3 sessions per week for 2 weeks, or twice monthly for 3 months.

Microservice is gaining increased adoption in the Telco NFV world. It is key to understand the design and deployment methodologies involved in developing Microservice based VNF. This talk provides an opensource practitioner approach to building and deploying a Microservice based VNF and includes the following: - Design patterns, workflow models - Design models for VNF placement, capacity management, scale-in/out and resiliency - Deployment considerations that includes handing of scale and fault tolerant VNF using well known Opensource tools. About the presenter: Prem Sankar works for Ericsson Opensource Ecosystem team and part of the Opendaylight and OPNFV team in Ericsson. Prem evangelizes SDN and Cloud and has given many sessions and conducted workshops around SDN and ODL. Prem is PTL of ODL COE project and currently driving the Kuberenetes and ODL Integration in Opendaylight community. Prem is a frequent speaker at opensource summits and has presented in Opendaylight, OPNFV and Open networking summits.

If you heard about web-scale or have a requirement to survive under web-scale or you just would like to prepare your application to handle an X effect this topic is for you. During a presentation you will understand aspects and caveats of performance testing, nuances of performance testing of Java based web applications. As a practical part you will get a brief overview of existing tools and will get a guide of using Gatling as a tool to make a load for your application. Gatling is an open source tool for performance loading written in Scala and provides comprehensive DSL for load scenario specification.