Presto anatomy

As we continue to push the boundaries of what is possible with respect to pipeline throughput and data serving tiers, new methodologies and techniques continue to emerge to handle larger and larger workloads

DataFusion is an extensible and embeddable query engine, written in Rust used to create modern, fast and efficient data pipelines, ETL processes, and database systems. This presentation explains where it fits into the data eco system and how it helps implement your system in Rust

The document discusses various techniques for profiling CPU and memory performance in Rust programs, including: - Using the flamegraph tool to profile CPU usage by sampling a running process and generating flame graphs. - Integrating pprof profiling into Rust programs to expose profiles over HTTP similar to how it works in Go. - Profiling heap usage by integrating jemalloc profiling and generating heap profiles on program exit. - Some challenges with profiling asynchronous Rust programs due to the lack of backtraces. The key takeaways are that there are crates like pprof-rs and techniques like jemalloc integration that allow collecting CPU and memory profiles from Rust programs, but profiling asynchronous programs

This document discusses using ClickHouse to manage log data. It begins with an introduction to ClickHouse and its features. It then covers different ways to model log data in ClickHouse, including storing logs as JSON blobs or converting them to a tabular format. The document demonstrates using materialized views to ingest logs into ClickHouse tables in an efficient manner, extracting values from JSON and converting to columns. It shows how this approach allows flexible querying of log data while scaling to large volumes.

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022 Apache Kafka without Zookeeper is now production ready! This talk is about how you can run without ZooKeeper, and why you should.

Oracle Database 19c, builds upon key architectural, distributed data and performance innovations established in earlier versions Oracle Database 12c and 18c releases. Oracle 19c has many new features, in this presentation we have covered below areas Automated Installation, Configuration and Patching AutoUpgrade and Database Utilities

The document provides an overview and agenda for a presentation on optimizing Oracle database performance through query tuning. It discusses identifying performance issues, collecting wait event information, reviewing execution plans, and understanding how the Oracle optimizer works using features like adaptive plans and statistics gathering. The goal is to show attendees how to quickly find and focus on the queries most in need of tuning.

This document discusses Kafka Streams state stores. It provides examples of using different types of windowing (tumbling, hopping, sliding, session) with state stores. It also covers configuring state store logging, caching, and retention policies. The document demonstrates how to define windowed state stores in Kafka Streams applications and discusses concepts like grace periods.

Presentation at Strata Data Conference 2018, New York The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives. In this tutorial, the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.

Presto is an open-source distributed SQL query engine for interactive analytics. It uses a connector architecture to query data across different data sources and formats in the same query. Presto's query planning and execution involves scanning data sources, optimizing query plans, distributing queries across workers, and aggregating results. Understanding Presto's query plans helps optimize queries and troubleshoot performance issues.

Getting Started with Elastic Stack. Detailed blog for the same http://vikshinde.blogspot.co.uk/2017/08/elastic-stack-introduction.html

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.

The document discusses how an easy-to-use and fast database can have a complicated implementation for developers. It outlines four key areas: 1) Flexible writing schema requires schema merging at read time. 2) Fast reads prune non-covered data chunks through predicate push-down. 3) Loading duplicated data necessitates data deduplication and compaction operations. 4) Quick data deletion still needs data elimination at read time or in the background. The document provides examples to illustrate the tradeoffs between user and developer requirements.

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

Presto generates Java bytecode at runtime to optimize query execution. Key query operations like filtering, projections, joins and aggregations are compiled into efficient Java methods using libraries like ASM and Fastutil. This bytecode generation improves performance by 30% through techniques like compiling row hashing for join lookups directly into machine instructions.

The document discusses the future of server-side JavaScript. It covers various Node.js frameworks and libraries that support both synchronous and asynchronous programming styles. CommonJS aims to provide interoperability across platforms by implementing synchronous proposals using fibers. Examples demonstrate how CommonJS allows for synchronous-like code while maintaining asynchronous behavior under the hood. Benchmarks show it has comparable performance to Node.js. The author advocates for toolkits over frameworks and continuing development of common standards and packages.

Node has captured the attention of early adopters by clearly differentiating itself as being asynchronous from the ground up while remaining accessible. Now that server side JavaScript is at the cutting edge of the asynchronous, real time web, it is in a much better position to establish itself as the go to language for also making synchronous, CRUD webapps and gain a stronger foothold on the server. This talk covers the current state of server side JavaScript beyond Node. It introduces Common Node, a synchronous CommonJS compatibility layer using node-fibers which bridges the gap between the different platforms. We look into Common Node's internals, compare its performance to that of other implementations such as RingoJS and go through some ideal use cases.

The power of Gatling is the DSL it provides to allow writing meaningful and expressive tests. We provide an overview of the framework, a description of their development environment and goals, and present their test results. Source code available https://github.com/lawlessc/random-response-time

The document introduces concepts related to Big Data technology including volume, variety, and velocity of data. It discusses Hadoop architecture including HDFS, MapReduce, YARN, and the Hadoop ecosystem. Examples are provided of common Big Data problems and how they can be solved using Hadoop frameworks like Pig, Hive, and Ambari.

This document provides an overview of CouchDB, a NoSQL database that uses JSON documents with a flexible schema. It demonstrates CouchDB's features like replication, MapReduce, and filtering. The presentation then shows how to build a mobile running app called Couch25K that tracks locations using CouchDB and syncs data between phones and a server. Code examples are provided in Objective-C, Java, and JavaScript for creating databases, saving documents, querying, and syncing.

This talk will begin with a discussion of the strengths of PostgreSQL and Hadoop. We will then lead into a high level overview of Hadoop and its community of projects like Hive, Flume and Sqoop. Finally, we will dig down into various use cases detailing how you can leverage Hadoop technologies for your PostgreSQL databases today. The use cases will range from using HDFS for simple database backups to using PostgreSQL and Foreign Data Wrappers to do low latency analytics on your Big Data.

Spark SQL enables Spark to perform efficient and fault-tolerant relational query processing with analytics database technologies. The relational queries are compiled to the executable physical plans consisting of transformations and actions on RDDs with the generated Java code. The code is compiled to Java bytecode, executed at runtime by JVM and optimized by JIT to native machine code at runtime. This talk will take a deep dive into Spark SQL execution engine. The talk includes pipelined execution, whole-stage code generation, UDF execution, memory management, vectorized readers, lineage based RDD transformation and action.

JavaScript is evolving with the addition of modules, platform consistency, and harmony features. Modules allow JavaScript code to be organized and avoid naming collisions. CommonJS and AMD module formats are used widely. Platform consistency is improved through polyfills that mimic future APIs for older browsers. Harmony brings language-level modules and features like destructuring assignment, default parameters, and promises to JavaScript. Traceur compiles Harmony code to existing JavaScript.

The document provides best practices for handling performance issues in an Odoo deployment. It recommends gathering deployment information, such as hardware specs, number of machines, and integration with web services. It also suggests monitoring tools to analyze system performance and important log details like CPU time, memory limits, and request processing times. The document further discusses optimizing PostgreSQL settings, using tools like pg_activity, pg_stat_statements, and pgbadger to analyze database queries and performance. It emphasizes reproducing issues, profiling code with tools like the Odoo profiler, and fixing problems in an iterative process.

Java EE, MicroProfile e Kotlin combinados para fazer microsserviços

This document provides an introduction and overview of a Node.js tutorial presented by Tom Hughes-Croucher. The tutorial covers topics such as building scalable server-side code with JavaScript using Node.js, debugging Node.js applications, using frameworks like Express.js, and best practices for deploying Node.js applications in production environments. The tutorial includes exercises for hands-on learning and demonstrates tools and techniques like Socket.io, clustering, error handling and using Redis with Node.js applications.

This talk was given at the Dutch PHP Conference 2011 and details the use of Comet (aka reverse ajax or ajax push) technologies and the importance of websockets and server-sent events. More information is available at http://joind.in/3237.

The document provides an overview of RxJava and its advantages over traditional Java streams and callbacks. It discusses key RxJava concepts like Observables, Observers, and Subscriptions. It demonstrates how to create Observables, subscribe to them, and compose operations like filter, map, and zip. It shows how to leverage schedulers to control threading. The document also provides examples of using RxJava with HTTP requests and the Twitter API to asynchronously retrieve user profiles and tweets. It highlights scenarios where RxJava is useful, like handling asynchronous operations, and discusses some pitfalls like its learning curve and need to understand backpressure.

This document introduces Reactive Programming with RxJava and how it can be used to create non-blocking applications. It discusses the limitations of blocking code and how RxJava uses Observables and Subscribers to implement reactive and asynchronous operations. It provides examples of converting blocking servlets and HTTP calls to non-blocking using RxJava. While non-blocking code is not always faster, it allows asynchronous operations to utilize threads more efficiently.

MongoDB is the trusted document store we turn to when we have tough data store problems to solve. For this talk we are going to go a little bit off the path and explore what other roles we can fit MongoDB into. Others have discussed how to turn MongoDB’s capped collections into a publish/subscribe server. We stretch that a little further and turn MongoDB into a full fledged broker with both publish/subscribe and queue semantics, and a the ability to mix them. We will provide code and a running demo of the queue producers and consumers. Next we will turn to coordination services: We will explore the fundamental features and show how to implement them using MongoDB as the storage engine. Again we will show the code and demo the coordination of multiple applications.

This 15 minute presentation discusses non-blocking I/O, event loops, and Node.js. It builds on previous work by Ryan Dahl, explaining how threads can be expensive due to context switching and memory usage, and how Node.js uses an event-driven, non-blocking model to avoid these costs. Code examples demonstrate getting and printing a policy object, handling HTTP requests asynchronously without blocking additional connections, and using callbacks to chain asynchronous actions together.

The document discusses Concurrency-oriented Programming (COP) using Erlang. It explains how Erlang programs work using lightweight processes that communicate asynchronously via message passing. This allows for high performance, reliability, and scalability. It provides examples of stateless server processes and using CouchDB for schema-free document storage accessible via REST APIs. Ruby libraries for interacting with CouchDB are also mentioned.

This document summarizes ql.io, a domain specific language for consuming HTTP APIs. Ql.io allows API calls to be made with fewer lines of code and reduced data sizes compared to traditional HTTP requests. It handles parallelizing requests and joining responses implicitly. Ql.io also allows mapping HTTP resources to SQL-like queries, enabling sequential and parallel queries over multiple APIs with a simple syntax. It can be used as an HTTP gateway or from Node.js.