High Performance Data Storage in a Microservices Environment

•

2 likes•996 views

This document discusses using microservices and in-memory data grids for high performance data storage and analytics. It shows how Apache Spark can be used for real-time analytics on data stored in an in-memory data grid. Examples are provided of SQL queries run on Spark to analyze user data and posts from a social network. The results are collected and written back to the data grid.

HIGH PERFORMANCE DATA STORAGE IN A
MICROSERVICES ENVIRONMENT

JBOSS DATA GRID IN-MEMORY DATA STORE
ACCELERATE BIG DATA ANALYTICS
7
In-memory data platform
BRMS Spark EAPHadoop
Spring
Boot
etc.
Cassandra
/ Red Hat
Storage
Data Overﬂow Mgmt.
to Persistence Store
(Hydration & Dehydration)
RDD/DStream Map-Reduce
Java Stream
In-
memory
Analytics
and
Compute
Grid
Application Frameworks
& APIs

GET VALUE OUT OF YOUR DATA
9
APACHE SPARK MICROSERVICES BUSINESS REPORTING

String query =
"SELECT displayName, reputation FROM users ORDER BY
reputation DESC LIMIT 10";

List<Row> rows = sqlContext.sql(query).collectAsList();
InfinispanJavaRDD.write(rows,properties);

String query =
"SELECT u.location as `Location`,count(p.id) as `Posts` "
+ "FROM users u "
+ "INNER JOIN posts p ON u.id=p.ownerUserId "
+ "WHERE p.postTypeId=1 "
+ "GROUP BY u.location ”
+ "ORDER BY `Posts` DESC LIMIT 10";

List<Row> rows = sqlContext.sql(query).collectAsList();

highestReputation
keyword
KeywordAnalytics
HighestRankAnalytics
DEMO DATA
PostStore
UserStore
owner = 2, dist_sync

plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
THANK YOU

Viewers also liked

Microservices with Docker, Kubernetes, and Jenkins

Red Hat Developers

By Rafael Benevides and Christian Posta A lot of functionality necessary for running in a microservices architecture have been built into Kubernetes; why would you re-invent the wheel with lots of complicated client-side libraries? Have you ever asked why you should use containers and what are the benefits for your application? This talk will present a microservices application that have been built using different Java platforms: WildFly Swarm and Vert.x. Then we will deploy this application in a Kubernetes cluster to present the advantages of containers for MSA (Microservices Architectures) and DevOps. The attendees will learn how to create, edit, build, deploy Java Microservices, and also how to perform service discovery, rolling updates, persistent volumes and much more. Finally we will fix a bug and see how a CI/CD Pipeline automates the process and reduces the deployment time.

Developing In Python On Red Hat Platforms (Nick Coghlan & Graham Dumpleton)

Red Hat Developers

Red Hat Software Collections, OpenShift and the Red Hat Container Development Kit open up many new possibilities for Python developers targeting Red Hat Enterprise Linux. At the same time, the wider Python ecosystem is undergoing two significant transitions - one being the ongoing migration from Python 2 to Python 3, and the other the shift to correctly validating HTTPS connections by default. In this session we will cover the currently available options for developing with Python on Red Hat platforms, as well as provide some insight into where things are headed in the context of the wider Python ecosystem.

Containers: Under The Hood (Vincent Batts)

Red Hat Developers

Despite the popularity and hype of containers, there is no need to regard containers as a block box. It is important to have an awareness of what's going on under the hood to help optimize your container requirements. In this session, we'll discuss: - Namespacing in the kernel - Copy-on-write storage choices - Portable container formats - Available container alternatives - Validation, trust, and content addressability with image verification See examples and options for your use-cases.

MicroServices for Java Developers

Red Hat Developers

The document discusses microservices for Java developers. It introduces Christian Posta, a principal middleware specialist and architect who works with large microservices and is a blogger and speaker on topics like DevOps, integration, and microservices. It then discusses how creating value through software is about speed, iteration, and continuous improvement. It covers concepts like distributed configuration, service discovery, load balancing, circuit breakers, and versioning/routing that are important for microservices. Finally, it mentions container cluster management with Kubernetes and technologies like Kubernetes, OpenShift, and Fabric8 that can help with microservices development.

Reactive Distributed Applications with Vert.x

Red Hat Developers

By Clement Escoffier Sorry, there is no free lunch—distributed applications are complex. You can embrace any trends such as microservices, but developing a distributed application is a challenge. Why? Distributed systems have many reasons to fail: technically they’re complicated, and the theory behind distributed systems is also complicated. Vert.x is a toolkit for building reactive distributed applications on top of the Java Virtual Machine in Java, JavaScript, Groovy, Ruby, or Ceylon. Vert.x does not hide the complexity of distributed applications; it lets you manage it. Vert.x applications are able to manage failures, can use several protocols and interaction styles, can handle heavy loads, and can cope with most of the requirements of modern applications.

CDK 2.0: Docker, Kubernetes, And OSE On Your Desk (Langdon White)

Red Hat Developers

Scale changes everything. What once was quite adequate for enterprise messaging can't scale to support "Internet of Things". We need new protocols, patterns and architectures to support this new world. This session will start with basic introduction to the concept of Internet of Things. Next it will discuss general technical challenges involved with the concept and explain why it is becoming mainstream now. Now we’re ready to start talking about solutions. We will introduce some messaging patterns (like telemetry and command/control) and protocols (such as MQTT and AMQP) used in these scenarios. Finally we will see how Apache ActiveMQ is gearing up for this race. We will show tips for horizontal and vertical scaling of the broker, related projects that can help with deployments and what the future development road map looks like.

Write Powerful Javascript Modules To Make Your Apps DRY (Brian Leathem)

Red Hat Developers

Large-scale Javascript applications benefit from a modular approach that let code be reused both within the application and across repeated implementations. In this session, we'll look at the modular approach used to build reusable Javascript modules in the Red Hat mobile field workforce management application (WFM) showcased in this year's Summit middleware keynote demo. Reusable modules for WFM are packaged as node package manager (npm) modules, providing a consistent format for both server and client sides using Node.js and Browserify. Modules are loosely coupled using the Mediator pattern and they broadcast user actions and state changes giving the application and other modules the opportunity to hook into those events. Additionally, visual components are packaged in a framework-agnostic manner, providing reusable UI components. You'll leave this session understanding the challenges faced when building reusable modules for large-scale applications, and the solutions employed in building out the reusable WFM modules.

Kubernetes for Java Developers

Red Hat Developers

By Rafael Benevides and Edson Yanaga Yes, Docker is great. We are all very aware of that, but now it’s time to take the next step: wrapping it all and deploying to a production environment. For this scenario, we need something more. For that “more,” we have Kubernetes by Google, a container platform based on the same technology used to deploy billions of containers per month on Google’s infrastructure. Ready to leverage your Docker skills and package your current Java app (WAR, EAR, or JAR)? Come to this session to see how your current Docker skill set can be easily mapped to Kubernetes concepts and commands. And get ready to deploy your containers in production.

Developer Meet Designer (Andres Galante & Brian Leathem)

Red Hat Developers

This presentation will take developers behind the scenes of the Keynote Demo to showcase how designers and a developers work together to achieve outstanding results. In this presentation, we'll identify the gap between designers and developers, and walk you through an actual example of how to build bridges that increase trust in your products. You'll learn about: - UX basics - Design within open source communities - Understanding the problems between developers and designers - The advantages (and disadvantages) of working with a designer - Coping with common pitfalls and false assumptions - Specific CSS and JS techniques used during the Keynote demo visualization You'll leave knowing that UX goes beyond the UI, with a better understanding of why working with a designer is important, and how to work together successfully.

Agile Is A Four-Letter Word (Jen Krieger)

Red Hat Developers

Based on a wide variety of surveys taken over recent years, many companies are transitioning to something that looks more like Agile than the processes they were using in previous years. However, that transition doesn’t necessarily mean implementations have been done respectfully of the Agile Manifesto and the principles behind it. In large part, industry trends seem to indicate that the sloganization of the word has done a significant disservice to the ideas that were originally founded in 2001. To add even more pain, most people seem to be entirely unaware of the core basis of Agile which is the idea to embrace change but inspect and adapt to that change. Are we lost as an industry? Is there anyway we can recover from this problem? In this session, attendees can expect to engage in a conversation about the rise of the Agile community, the negative and positive impact it has had on the industry, and how you individually can help your organizations and teams lower the risk of encountering the negative problems, and speed your way towards the positives. Topics will include: - The intentions behind agile - Ways you can rework or improve your not so great agile situation - Things you should avoid from the start.

Full Stack Development With Node.Js And NoSQL (Nic Raboy & Arun Gupta)

Red Hat Developers

In this session, we'll talk about what's different about this generation of web applications and how a solid development approach must consider the latency, throughput, and interactivity demand by users across mobile devices, web browsers, and Internet of Things (IoT). We'll demonstrate how to include Couchbase in such applications to support a flexible data model and the easy scalability required for modern development. We'ill demonstrate how to create a full stack application focusing on the CEAN stack, which is composed of Couchbase, Express Framework, AngularJS, and Node.js.

Building Reactive Applications With Node.Js And Red Hat JBoss Data Grid (Gald...

Red Hat Developers

Node.js is a very popular framework for developing asynchronous, event-driven, reactive applications. Red Hat JBoss Data Grid, an in-memory distributed database designed for fast access to large volumes of data and scalability, has recently gained compatibility with Node.js letting reactive applications use it as a persistence layer. Thanks to near caching, JBoss Data Grid offers excellent response times for data queried regularly, and its continuous remote event support means data can get pushed from the data grid to the Node.js application instead of having to wait for the data grid to serve it. In this session, we'll show how to build Node.js applications that use JBoss Data Grid as a persistence layer.

Boost Development With Java EE7 On EAP7 (Demitris Andreadis)

Red Hat Developers

DevOps Moves To Production (Lori MacVittie)

Red Hat Developers

Delivering an app or service fast and frequently to production isn't the same as delivering the app or service fast and frequently to its intended users. Before an app is actually 'live' it has to run the gauntlet of production deployment that stands between it and real, live users. While DevOps has helped organizations make huge strides toward continuous delivery in dev and test environments, the production environment remains a very real obstacle in realizing continuous deployment. The biggest hurdle in that obstacle course is a narrow definition of DevOps that fails to include a broad set of technologies and tools outside the Dev and Ops domain. In this session we'll explore the underlying elements of a comprehensive DevOps approach (SDN, CD/CI, and Agile) and how they mix, match, and combine to enable the operational transformation DevOps promises to achieve the ultimate goal of IT agility: continuous deployment.

Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)

Red Hat Developers

Just like a spoon full of sugar will cure your hiccups, running your JVM with -XX:+UseShenandoahGC will cure your Java garbage collection hiccups. Shenandoah GC is a new garbage collector algorithm developed for OpenJDK at Red Hat, which will produce much better pause times than the currently-available algorithms without a significant decrease in throughput. In this session, we'll explain how Shenandoah works and compare it to the currently-available OpenJDK garbage collectors.

Java and DevOps: Supercharge Your Delivery Pipeline with Containers

Red Hat Developers

How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...

Red Hat Developers

The fifth major release of Hibernate sports contains many internal changes developed in collaboration between the Hibernate team and the Red Hat middleware performance team. Efficient access to databases is crucial to get scalable and responsive applications. Hibernate 5 received much attention in this area. You’ll benefit from many of these improvements by merely upgrading. But it's important to understand some of these new, performance-boosting features because you will need to explicitly enable them. We'll explain the development background on all of these powerful new features and the investigation process for performance improvements. Our aim is to provide good guidance so you can make the most of it on your own applications. We'll also peek at other performance improvements made on JBoss EAP 7, like on the caching layer, the connection manager, and the web tier. We want to make sure you can all enjoy better-performing applications—that require less power and less servers—without compromising on your developer’s productivity.

The Deploy Master: From Basic to Zero Downtime, Blue/Green, A/B and Canary

Red Hat Developers

By Edson Yanaga and Rafael Benevides The “deploy moment” is an occasion that still gives many developers the shivers. But it shouldn’t be this way (at least not every time). Luckily enough, we have tools and processes today that enable us to turn the deploy moment into a usual activity. Check out this session to learn how we can evolve our Java deployment process from the very basic to zero downtime and then apply some very interesting strategies such as blue/green, A/B, and Canary deployments.

Ultimate DevOps: OpenShift Dedicated With CloudBees Jenkins Platform (Andy Pe...

Red Hat Developers

Are you ready to innovate with cloud-native app development? Are you ready to accelerate business agility with continuous delivery (CD)? Well, now you can easily do both using CloudBees Jenkins Platform within OpenShift Dedicated by Red Hat. In this session, you'll learn how to seamlessly use this CD solution to fully automate your application development, test, and delivery life cycle. Using the CloudBees platform to automate your CD pipelines allows your developers to focus on what they do best—innovating. Combine that with the elasticity and scale of the Docker-based OpenShift Dedicated environment, and you'll remove many of the obstacles to business growth. Come see the future of digital innovation.

Architecting cloud-enabled applications using Spring-Integration 2.x

Edson Yanaga

Viewers also liked (20)

Microservices with Docker, Kubernetes, and Jenkins

Developing In Python On Red Hat Platforms (Nick Coghlan & Graham Dumpleton)

Containers: Under The Hood (Vincent Batts)

MicroServices for Java Developers

Reactive Distributed Applications with Vert.x

CDK 2.0: Docker, Kubernetes, And OSE On Your Desk (Langdon White)

Write Powerful Javascript Modules To Make Your Apps DRY (Brian Leathem)

Kubernetes for Java Developers

Developer Meet Designer (Andres Galante & Brian Leathem)

Agile Is A Four-Letter Word (Jen Krieger)

Full Stack Development With Node.Js And NoSQL (Nic Raboy & Arun Gupta)

Building Reactive Applications With Node.Js And Red Hat JBoss Data Grid (Gald...

Boost Development With Java EE7 On EAP7 (Demitris Andreadis)

DevOps Moves To Production (Lori MacVittie)

Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)

Java and DevOps: Supercharge Your Delivery Pipeline with Containers

How To Get The Most Out Of Your Hibernate, JBoss EAP 7 Application (Ståle Ped...

The Deploy Master: From Basic to Zero Downtime, Blue/Green, A/B and Canary

Ultimate DevOps: OpenShift Dedicated With CloudBees Jenkins Platform (Andy Pe...

Architecting cloud-enabled applications using Spring-Integration 2.x

Similar to High Performance Data Storage in a Microservices Environment

Big Data on the Cloud

Sercan Karaoglu

This document discusses big data processing on the cloud. It describes the 3V model of big data, which refers to volume, velocity and variety. It discusses operational challenges of big data and how distributed processing tools like Hadoop and Spark can help address these challenges. It provides an example use case of using Apache Spark on Amazon Web Services along with Apache Zeppelin and Amazon S3 for distributed data analytics.

SAP HANA SPS09 - XS Programming Model

SAP Technology

The document discusses new features in SAP HANA SPS 09 for the XS programming model, including improvements to security, user handling, outbound connectivity, the database interface, core XSJS APIs, and more. Key highlights include full CORS support, secure HTTP session cookies, a new lightweight database interface with better performance and usability, new XSJS APIs for SMTP, ZIP/GZIP, and asynchronous requests, and enhanced user handling options.

SparkR-Advance Analytic for Big Data

samuel shamiri

This document outlines an agenda for a workshop on SparkR, which is an interface for R that allows users to leverage Apache Spark's distributed processing capabilities. The agenda includes an introduction to Spark and DataFrames, an overview of SparkR, and a machine learning demo. It also provides background on the speaker and his company, which uses data analytics to help businesses make decisions. Spark can process diverse data formats across various launch modes and platforms, and SparkR allows R users to take advantage of Spark's performance benefits while retaining R's interactive environment.

Machine Learning with H2O, Spark, and Python at Strata 2015

Sri Ambati

TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...

Debraj GuhaThakurta

TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...

Debraj GuhaThakurta

R is a popular statistical programming language used for data analysis and machine learning. It has over 3 million users and is taught widely in universities. While powerful, R has some scaling limitations for big data. Several Apache Spark integrations with R like SparkR and sparklyr enable distributed, parallel processing of large datasets using R on Spark clusters. Other options for scaling R include H2O for in-memory analytics, Microsoft ML Server for on-premises scaling, and ScaleR for portable parallel processing across platforms. These solutions allow R programs and models to be trained on large datasets and deployed for operational use on big data in various cloud and on-premises environments.

IMC Summit 2016 Breakout - William Bain - Implementing Extensible Data Struct...

In-Memory Computing Summit

In-memory data grids (IMDGs) are widely used as distributed, key-value stores for serialized objects, providing fast data access, location transparency, scalability, and high availability. With its support for built-in data structures, such as hashed sets and lists, Redis has demonstrated the value of enhancing standard create/read/update/delete (CRUD) APIs to provide extended functionality and performance gains. This talk describes new techniques which can be used to generalize this concept and enable the straightforward creation of arbitrary, user-defined data structures both within single objects and sharded across the IMDG. A key challenge for IMDGs is to minimize network traffic when accessing and updating stored data. Standard CRUD APIs place the burden of implementing data structures on the client and require that full objects move between client and server on every operation. In contrast, implementing data structures within the server streamlines communication since only incremental changes to stored objects or requested subsets of this data need to be transferred. However, building extended data structures within IMDG servers creates several challenges, including, how to extend this mechanism, how to efficiently implement data-parallel operations spanning multiple shards, and how to protect the IMDG from errors in user-defined extensions. This talk will describe two techniques which enable IMDGs to be extended to implement user-defined data structures. One technique, called single method invocation (SMI), allows users to define a class which implements a user-defined data structure stored as an IMDG object and then remotely execute a set of class methods within the IMDG. This enables IMDG clients to pass parameters to the IMDG and receive a result from method execution. A second technique, called parallel method invocation (PMI), extends this approach to execute a method in parallel on multiple objects sharded across IMDG servers. PMI also provides an efficient mechanism for combining the results of method execution and returning a single result to the invoking client. In contrast to client-based techniques, this combining mechanism is integrated into the IMDG and completes in O(logN) time, where N is the number of IMDG servers. The talk will describe how user-defined data structures can be implemented within the IMDG to run in a separate process (e.g., a JVM) to ensure that execution errors do not impair the stability of the IMDG. It will examine the associated performance trade-offs and techniques that can be used to minimize overhead. Lastly, the talk will describe how popular Redis data structures, such as hashed sets, can be implemented as a user-defined data structure using SMI and then extended using both SMI and PMI to build a scalable hashed set that spans multiple shards. It will also examine other examples of user-defined data structures that can be built using these techniques.

Using your DB2 SQL Skills with Hadoop and Spark

Cynthia Saracco

Lessons from the Field, Episode II: Applying Best Practices to Your Apache S...

Databricks

Apache Spark is an excellent tool to accelerate your analytics, whether you’re doing ETL, Machine Learning, or Data Warehousing. However, to really make the most of Spark it pays to understand best practices for data storage, file formats, and query optimization. As a follow-up of last year’s “Lessons From The Field”, this session will review some common anti-patterns I’ve seen in the field that could introduce performance or stability issues to your Spark jobs. We’ll look at ways of better understanding your Spark jobs and identifying solutions to these anti-patterns to help you write better performing and more stable applications.

Big Data: SQL query federation for Hadoop and RDBMS data

Cynthia Saracco

Dancing with the Elephant

DataWorks Summit

The document discusses using Teradata's Unified Data Architecture and SQL-MapReduce functions to analyze customer churn for a telecommunications company. It provides examples of creating views that join customer data from Teradata, Hadoop, and Aster sources. Graphing and visualization tools are used to identify patterns in customer reboot events and equipment issues that may lead to cancellations. The document demonstrates how to gain insights into customer behavior across multiple data platforms.

Thu-310pm-Impetus-SachinAndAjay

Ajay Shriwastava

This document discusses the emergence of logical data warehouses and how they can help organizations address challenges posed by big data. A logical data warehouse takes a virtualized approach to integrating data from multiple sources like relational databases, NoSQL stores, and file systems. It provides a single, unified view of data while keeping the underlying systems decoupled. The document also describes how organizations can use techniques like data virtualization and offloading to optimize workloads between their enterprise data warehouse and Hadoop data lake. This helps reduce costs while improving query performance and resource utilization.

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

Databricks

This introductory workshop is aimed at data analysts & data engineers new to Apache Spark and exposes them how to analyze big data with Spark SQL and DataFrames. In this partly instructor-led and self-paced labs, we will cover Spark concepts and you’ll do labs for Spark SQL and DataFrames in Databricks Community Edition. Toward the end, you’ll get a glimpse into newly minted Databricks Developer Certification for Apache Spark: what to expect & how to prepare for it. * Apache Spark Basics & Architecture * Spark SQL * DataFrames * Brief Overview of Databricks Certified Developer for Apache Spark

Logical Data Warehouse: How to Build a Virtualized Data Services Layer

DataWorks Summit

The document discusses the emergence of logical data warehouses in response to big data. It describes how a logical data warehouse uses virtualization, distributed processing, and other techniques to provide a unified view of data across different repositories like Hadoop, relational databases and NoSQL stores. It also discusses how organizations can optimize resources by offloading analytical workloads from their enterprise data warehouse to Hadoop clusters to reduce costs while still using existing code and applications.

Building highly scalable data pipelines with Apache Spark

Martin Toshev

This document provides a summary of Apache Spark, including: - Spark is a framework for large-scale data processing across clusters that is faster than Hadoop by relying more on RAM and minimizing disk IO. - Spark transformations operate on resilient distributed datasets (RDDs) to manipulate data, while actions return results to the driver program. - Spark can receive data from various sources like files, databases, sockets through its datasource APIs and process both batch and streaming data. - Spark streaming divides streaming data into micro-batches called DStreams and integrates with messaging systems like Kafka. Structured streaming is a newer API that works on DataFrames/Datasets.

AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...

Amazon Web Services

Amazon Machine Learning is a service that makes it easy for developers of all skill levels to use machine learning technology and Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools. The combination of the two can provide a solution to power advanced analytics for not only what has happened in the past, but make intelligent predictions about the future. Please join this webinar to learn how get the most value from your data for your data driven business. Learning Objectives: How to scale your Redshift queries with user-defined functions (UDFs) How to apply Machine learning to historical data in Amazon Redshift How to visualize your data with Amazon QuickSight Present a reference architecture for advanced analytics Who Should Attend: Application developers looking to add UDFs, or predictive analytics to their applications, database administrators that need to meet the demand of data driven organizations, decision makers looking to derive more insight from their data

Microsoft R Server for Data Sciencea

Data Science Thailand

Spark + Cassandra = Real Time Analytics on Operational Data

Victor Coustenoble

This document discusses using Apache Spark and Cassandra together for real-time analytics on transactional data. It provides an overview of Cassandra and how it can be used for operational applications like recommendations, fraud detection, and messaging. It then discusses how the Spark Cassandra Connector allows reading and writing Cassandra data from Spark, enabling real-time analytics on streaming and batch data using Spark SQL, MLlib, and Spark Streaming. It also covers some DataStax Enterprise features for high availability and integration of Spark and Cassandra.

Big Data: SQL on Hadoop from IBM

Cynthia Saracco

Big SQL provides an SQL interface for querying data stored in Hadoop. It uses a new query engine derived from IBM's database technology to optimize queries. Big SQL allows SQL users easy access to Hadoop data through familiar SQL tools and syntax. It supports creating and loading tables, standard SQL queries including joins and subqueries, and integrating Hadoop data with external databases in a single query.

Accelerate and modernize your data pipelines

Paul Van Siclen

In this session you will learn how Qlik’s Data Integration platform (formerly Attunity) reduces time to market and time to insights for modern data architectures through real-time automated pipelines for data warehouse and data lake initiatives. Hear how pipeline automation has impacted large financial services organizations ability to rapidly deliver value and see how to build an automated near real-time pipeline to efficiently load and transform data into a Snowflake data warehouse on AWS in under 10 minutes.

Similar to High Performance Data Storage in a Microservices Environment (20)

Big Data on the Cloud

SAP HANA SPS09 - XS Programming Model

SparkR-Advance Analytic for Big Data

Machine Learning with H2O, Spark, and Python at Strata 2015

TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...

TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...

IMC Summit 2016 Breakout - William Bain - Implementing Extensible Data Struct...

Using your DB2 SQL Skills with Hadoop and Spark

Lessons from the Field, Episode II: Applying Best Practices to Your Apache S...

Big Data: SQL query federation for Hadoop and RDBMS data

Dancing with the Elephant

Thu-310pm-Impetus-SachinAndAjay

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

Logical Data Warehouse: How to Build a Virtualized Data Services Layer

Building highly scalable data pipelines with Apache Spark

AWS November Webinar Series - Advanced Analytics with Amazon Redshift and the...

Microsoft R Server for Data Sciencea

Spark + Cassandra = Real Time Analytics on Operational Data

Big Data: SQL on Hadoop from IBM

Accelerate and modernize your data pipelines

More from Red Hat Developers

DevNation Tech Talk: Getting GitOps

Red Hat Developers

In 2022 we heard your GitOps questions at meetups and gatherings, big stages and local panels and one question was often top of mind: how do I get started? The benefits of GitOps are calling your name, but getting started isn’t that straightforward. Red Hat is excited to kick off 2023 with a DevNation TechTalk, focused on GitOps to help you sift through your questions. At DevNation you’ll hear from passionate GitOps practitioners about the pitfalls to avoid and hurdles to jump while kicking off or evolving your GitOps practices. This event is aimed at audiences that are new to GitOps or early in their practice development within a cloud native environment. During this live session you’ll learn: Upcoming updates and key milestones in the ArgoCD roadmap and how Red Hat will support them How to simplify the delivery GitOps across multi-cloud environments GitOps best practices from experts at: PostNord Strålfors: Filip Jansson Arbetsförmedlingen: Misho Kmetovski & Richard Hermansson Swiss Railways (SBB): Manuel Wallrapp & Thomas Bruederli Plus stick around for an “Ask me Anything” segment to ask any outstanding questions live.

Exploring the power of OpenTelemetry on Kubernetes

Red Hat Developers

Modern cloud-native applications are incredibly complex systems. Keeping the systems healthy and meeting SLAs for our customers is crucial for long-term success. In this session, we will dive into the three pillars of observability - metrics, logs, tracing - the foundation of successful troubleshooting in distributed systems. You'll learn the gotchas and pitfalls of rolling out the OpenTelemetry stack on Kubernetes to effectively collect all your signals without worrying about a vendor lock in. Additionally we will replace parts of the Prometheus stack to scrape metrics with OpenTelemetry collector and operator.

GitHub Makeover | DevNation Tech Talk

Red Hat Developers

GitHub plays a key role in the everyday work of thousands of developers and is a central piece of the open-source software ecosystem. Even though it is getting better and better every day, it still misses some key features that we need. If you want a better way of reviewing PRs, navigating through the code or better yet - writing the code without leaving the browser - this talk is for you! This talk will be demo driven, and as the title suggests, we will start with the aesthetic revamp. But we definitely won’t stop there! You will also learn a few cool things about interacting with GitHub through the command line. So not only your UI will be officially revamped, but you will also gain a productivity boost.

Quinoa: A modern Quarkus UI with no hassles | DevNation tech Talk

Red Hat Developers

Extra micrometer practices with Quarkus | DevNation Tech Talk

Red Hat Developers

This document discusses using metrics to monitor Quarkus applications. It recommends metrics like throughput, memory usage, queue time, average response time, and error rates. It explains how Quarkus supports Micrometer for instrumenting applications with metrics and integrating with monitoring systems. The document includes a demo of adding metrics to code. It provides tips for using annotations and tags to gain more insights from metrics. Source code examples are linked.

Event-driven autoscaling through KEDA and Knative Integration | DevNation Tec...

Red Hat Developers

This talk will teach you how to redesign an event-driven autoscaling architecture for cloud-native microservices by utilizing Apache Kafka, Knative, and KEDA infrastructure. You will also learn how to deploy serverless applications (Quarkus) using a Knative service. Finally, KEDA will enable you to autoscale Knative Eventing components (KafkaSource) through events consumption over standard resources (CPU, memory).

Integrating Loom in Quarkus | DevNation Tech Talk

Red Hat Developers

Loom is among the most highly anticipated projects in the Java world. It promises to address concurrency and Java execution model issues by providing virtual threads. Thus, there is no need to write concurrent programs using asynchronous or reactive APIs; it will be possible to use the traditional imperative model and let Loom handle the rest. The JVM will execute the program and leverage non-blocking APIs automatically! Sounds good, doesn't it? How does it work, though? Are there any hidden costs? What is Loom going to change in modern Java frameworks? We will answer these questions in this talk. Starting with the integration of Loom in Quarkus, we will compare the different approaches we considered, discuss their respective pros and cons, and show how Loom might change the Java world.

Quarkus Renarde 🦊♥: an old-school Web framework with today's touch | DevNatio...

Red Hat Developers

Quarkus Renarde 🦊♥ is a new Web framework based on Quarkus. This framework focuses not on microservices but web applications and makes Quarkus even easier to use for web apps: - Endpoints based on convention, even easier than RESTEasy Reactive and JAX-RS - Server-side templating with Qute - Validation with Hibernate Validation - Data with Hibernate ORM or Reactive with Panache - Simple authentication with OpenID Connect or WebAuthn Quarkus Renarde 🦊♥ can deliver all this while still providing the joy of developing with Quarkus, with live reload, continuous testing, the Dev, and more.

Containers without docker | DevNation Tech Talk

Red Hat Developers

This document summarizes a talk about running containers without Docker. It discusses alternatives like Podman and Buildah that can replace Docker functionality. The talk demonstrates installing and using Podman to run containers, Buildah to build images from Dockerfiles, and Skopeo to copy images between registries. The presentation encourages understanding containers beyond just Docker and knowing other tools in the ecosystem.

Distributed deployment of microservices across multiple OpenShift clusters | ...

Red Hat Developers

Hybrid-cloud and multi-cloud patterns are the next application deployment architectures, and Kubernetes is the de facto container orchestration engine. 50% of production Kubernetes workloads involve some form of microservices applications. How can we manage this inter-cluster application connectivity? Meet Skupper: an open-source project that solves multi-cloud communication for Kubernetes. In this Tech Talk, you will briefly learn about Skupper and watch a live demo of an e-commerce application with 10 microservices spanning three OpenShift clusters running on three different public cloud providers.

DevNation Workshop: Object detection with Red Hat OpenShift Data Science [Mar...

Red Hat Developers

In this workshop, you’ll learn an easy way to incorporate data science and AI/ML into an OpenShift development workflow. As an example, you’ll use an object detection model to detect ‘dog(s)’ in an image. You will: Use Jupyter Notebooks and TensorFlow to explore a pre-trained object detection model Serve the model in a REST API as a Flask App Use Source-to-Image (S2I) to build and deploy the Flask app Explore Kafka streams from Notebooks Deploy a Kafka consumer with the same object detection model You’ll be able to do all of this without having to install anything on your own computer, thanks to Red Hat OpenShift Data Science and Red Hat OpenShift Streams for Apache Kafka. Note: Beginner data handling and Python skills are required for this workshop.

Dear security, compliance, and auditing: We’re sorry. Love, DevOps | DevNatio...

Red Hat Developers

DevOps solved the conflict between development and operations, but other essential aspects of the delivery lifecycle—security, compliance, and audit—were left out. DevSecOps is an excellent reminder that security must be DevOps’d, but compliance and audit are still missing. There’s no need for a new DevSecAuditComplianceOps buzzword; instead, let’s talk about continuous authorization, which applies Zero Trust principles to continuous monitoring. In this tech talk, Bill Bensing will discuss practical ways to start with continuous authorization for the software delivery lifecycle using Ploigos.

11 CLI tools every developer should know | DevNation Tech Talk

Red Hat Developers

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

GitHub Actions and OpenShift: Supercharging your software development loops...

Red Hat Developers

Every software developer wants more productivity. What if the only commands you needed to deploy were "git commit" and "git push"? Join us as we walk you through a live demonstration of how you can ship your lovely application code from your local machine to a free OpenShift cluster, fully automated through GitHub Actions. By the end of this session, you'll have a sound understanding of building a GitHub Action workflow for your codebase that leverages OpenShift to deploy your application.

To the moon and beyond with Java 17 APIs! | DevNation Tech Talk

Red Hat Developers

Since moving to a 6 monthly release cadence, the Java platform is evolving more dynamically than ever before. It can be quite a challenge to stay on top of all the changes and new features. In this talk we're going to explore the most important developments in the Java API: which classes have been added, and what has been removed? Join Duke, the Java mascot, for a trip to space and learn which exciting new APIs provided by the Java platform can help him with his journey: The Java Vector API for utilizing the SIMD capabilities of modern CPU architectures The Foreign Linker API for integrating with native code The JFR Event Streaming API for publishing JDK Flight Recorder Events We'll also take a look at some useful changes to the Java runtime, such as CDS archives for a faster spaceship..., uhm, application launch!

Profile your Java apps in production on Red Hat OpenShift with Cryostat | Dev...

Red Hat Developers

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

Red Hat Developers

This document discusses Apache Kafka and Red Hat OpenShift Streams for Apache Kafka. It begins with an overview of what Apache Kafka is and its common use cases. It then demonstrates how Red Hat OpenShift Streams provides a managed Apache Kafka cluster as a service, including a dedicated cluster, configuration management, metrics, monitoring and other features to provide a streamlined developer experience. It concludes with information on trying OpenShift Streams for Apache Kafka and additional resources.

Kubernetes configuration and security policies with KubeLinter | DevNation Te...

Red Hat Developers

With Kubernetes, implementing security policies can be challenging. First, developers, administrators, and security teams need to understand security policies in collaboration to have the best chance of successful adoption. Next, policy enforcement needs to integrate with developer workflows. Lastly, policies need to contain corrective action that is as close to the developer as possible. KubeLinter solves these problems by linting Kubernetes YAML files and Helm charts at the source: the developer. In this session, we will evaluate KubeLinter by moving through a hands-on demo of the application, showing a use case for local machines and CI pipeline integration, and chatting about how best to integrate it into your organization: KubeLinter, and its default checks How you can leverage the application in your day-to-day operations The open source StackRox community

Level-up your gaming telemetry using Kafka Streams | DevNation Tech Talk

Red Hat Developers

Many modern video games are constantly evolving post-release. New maps, game modes, and game balancing adjustments are rolled out, often on a weekly basis. This continuous iteration to improve player engagement and satisfaction requires data-driven decision making based on events and telemetry captured during gameplay, and from community forums and discussions. In this session you will learn how OpenShift Streams for Apache Kafka and Kafka Streams can be used to analyze real-time events and telemetry reported by a game server, using a practical example that encourages audience participation. Specifically you’ll learn how to: Provision Kafka clusters on OpenShift Streams for Apache Kafka. Develop a Java application that uses Kafka Streams and Quarkus to process event data. Deploy the application locally, or on OpenShift and connect it to your OpenShift Streams for Apache Kafka Cluster.

More from Red Hat Developers (20)