Data Analytics Service Company and Its Ruby Usage

This document discusses automating analytics pipelines and workflows using a workflow engine. It describes the challenges of managing workflows across multiple cloud services and database technologies. It then introduces a multi-cloud workflow engine called Digdag that can automate workflows, handle errors, enable parallel execution, support modularization and parameterization. Examples are given of using Digdag to define and run workflows across services like BigQuery, Treasure Data, Redshift, and Tableau. Key features of Digdag like loops, parameters, parallel tasks and pushing workflows to servers with Docker are also summarized.

This document discusses middleware in Ruby and provides examples of considerations when writing middleware: - Middleware should be a long-running daemon process that is compatible across platforms and environments and handles various data formats and traffic volumes. - Tests must be run on all supported platforms to ensure compatibility as thread and process scheduling differs between operating systems. - Memory usage and object leaks must be carefully managed in long-running processes to avoid consuming resources over time. - Performance of JSON parsing/generation should be benchmarked and the most optimized library used to avoid unnecessary CPU usage.

1) The document proposes making a key-value storage system (CDP KVS) 10 times more scalable to support real-time data delivery. 2) Three ideas are presented: using an alternative distributed KVS, implementing a storage hierarchy on the existing KVS, and shipping edit logs to indexed archives. 3) The storage hierarchy approach of partitioning, compressing, and writing data to DynamoDB in batches is selected as it improves write performance and reduces storage costs while remaining stateless.

This document summarizes Tagomori Satoshi's presentation on handling "not so big data" at the YAPC::Asia 2014 conference. It discusses different types of data processing frameworks for various data sizes, from sub-gigabytes up to petabytes. It provides overviews of MapReduce, Spark, Tez, and stream processing frameworks. It also discusses what Hadoop is and how the Hadoop ecosystem has evolved to include these additional frameworks.

Digdag can automate large-scale data processing and handle errors. It provides constructs like operators, parameters, and task groups to organize workflows. Operators package tasks to run queries or process data. Parameters allow passing variables between tasks. Task groups modularize and organize workflows. Digdag supports error handling, monitoring, parallelization, versioning, and reproducing workflows across environments.

Fluentd is an open source data collector that allows flexible data collection, processing, and output. It supports streaming data from sources like logs and metrics to destinations like databases, search engines, and object stores. Fluentd's plugin-based architecture allows it to support a wide variety of use cases. Recent versions of Fluentd have added features like improved plugin APIs, nanosecond time resolution, and Windows support to make it more suitable for containerized environments and low-latency applications.

Apache Spark is one of the most popular big data projects, offering greatly improved performance over traditional MapReduce models. Much of Apache Spark’s power comes from lazy evaluation along with intelligent pipelining, which can make debugging more challenging. Holden Karau and Joey Echeverria explore how to debug Apache Spark applications, the different options for logging in Spark’s variety of supported languages, and some common errors and how to detect them. Spark’s own internal logging can often be quite verbose. Holden and Joey demonstrate how to effectively search logs from Apache Spark to spot common problems and discuss options for logging from within your program itself. Spark’s accumulators have gotten a bad rap because of how they interact in the event of cache misses or partial recomputes, but Holden and Joey look at how to effectively use Spark’s current accumulators for debugging before gazing into the future to see the data property type accumulators that may be coming to Spark in future versions. And in addition to reading logs and instrumenting your program with accumulators, Spark’s UI can be of great help for quickly detecting certain types of problems. Holden and Joey cover how to quickly use the UI to figure out if certain types of issues are occurring in your job. The talk will wrap up with Holden trying to get everyone to buy several copies of her new book, High Performance Spark.

This document provides tips for troubleshooting common issues with Sqoop. It discusses how to effectively provide debugging information when seeking help, addresses specific problems with MySQL connections and importing to Hive, Oracle case-sensitive errors and export failures, and recommends best practices like using separate tables for import and export and specifying options correctly.

Tomas Doran presented on their implementation of Logstash at TIM Group to process over 55 million messages per day. Their applications are all Java/Scala/Clojure and they developed their own library to send structured log events as JSON to Logstash using ZeroMQ for reliability. They index data in Elasticsearch and use it for metrics, alerts and dashboards but face challenges with data growth.

User defined partitioning is a new partitioning strategy in Treasure Data that allows users to specify which column to use for partitioning, in addition to the default "time" column. This provides more flexible partitioning that better fits customer data platform workloads. The user can define partitioning rules through Presto or Hive to improve query performance by enabling colocated joins and filtering data by the partitioning column.

Apache Spark is one of the most popular big data projects, offering greatly improved performance over traditional MapReduce models. Much of Apache Spark’s power comes from lazy evaluation along with intelligent pipelining, which can make debugging more challenging. This talk will examine how to debug Apache Spark applications, the different options for logging in Spark’s variety of supported languages, as well as some common errors and how to detect them. Spark’s own internal logging can often be quite verbose, and this talk will examine how to effectively search logs from Apache Spark to spot common problems. In addition to the internal logging, this talk will look at options for logging from within our program itself. Spark’s accumulators have gotten a bad rap because of how they interact in the event of cache misses or partial recomputes, but this talk will look at how to effectively use Spark’s current accumulators for debugging as well as a look to future for data property type accumulators which may be coming to Spark in future version. In addition to reading logs, and instrumenting our program with accumulators, Spark’s UI can be of great help for quickly detecting certain types of problems.

An updated talk about how to use Solr for logs and other time-series data, like metrics and social media. In 2016, Solr, its ecosystem, and the operating systems it runs on have evolved quite a lot, so we can now show new techniques to scale and new knobs to tune. We'll start by looking at how to scale SolrCloud through a hybrid approach using a combination of time- and size-based indices, and also how to divide the cluster in tiers in order to handle the potentially spiky load in real-time. Then, we'll look at tuning individual nodes. We'll cover everything from commits, buffers, merge policies and doc values to OS settings like disk scheduler, SSD caching, and huge pages. Finally, we'll take a look at the pipeline of getting the logs to Solr and how to make it fast and reliable: where should buffers live, which protocols to use, where should the heavy processing be done (like parsing unstructured data), and which tools from the ecosystem can help.

Cloudera Morphlines is a new open source framework, recently added to the CDK, that reduces the time and skills necessary to integrate, build, and change Hadoop processing applications that extract, transform, and load data into Apache Solr, Apache HBase, HDFS, enterprise data warehouses, or analytic online dashboards.

The document discusses Azure Cache and Redis. It provides an overview of Redis, including its data types, transactions, pub/sub capabilities, scripting, and sharding/partitioning. It then discusses common patterns for using Redis, such as caching, counting likes on Facebook, getting the latest reviews, rate limiting, and autocompletion. The document emphasizes that Redis is very flexible and can be used for more than just caching, acting as a general datastore. It concludes by recommending a Redis reference book for further learning.

Using Solr to Search and Analyze Logs discusses how to use Solr and related tools like Logstash and Elasticsearch to index, search, and analyze log data. It covers sending logs to Solr using various methods like Logstash, Flume, and rsyslog plugins. It also discusses best practices for handling logs like production data in Solr, including using docValues, omitting norms/term positions, configuring caches, commits, and merges. The document concludes by discussing scaling options like SolrCloud and collections/aliases for partitioning logs over time.

Presented by Rahul Jain, System Analyst (Software Engineer), IVY Comptech Pvt Ltd Consolidation and Indexing of logs to search them in real time poses an array of challenges when you have hundreds of servers producing terabytes of logs every day. Since the log events mostly have a small size of around 200 bytes to few KBs, makes it more difficult to handle because lesser the size of a log event, more the number of documents to index. In this session, we will discuss the challenges faced by us and solutions developed to overcome them. The list of items that will be covered in the talk are as follows. Methods to collect logs in real time. How Lucene was tuned to achieve an indexing rate of 1 GB in 46 seconds Tips and techniques incorporated/used to manage distributed index generation and search on multiple shards How choosing a layer based partition strategy helped us to bring down the search response times. Log analysis and generation of analytics using Solr. Design and architecture used to build the search platform.

Cloud deployments of Apache Hadoop are becoming more commonplace. Yet Hadoop and it's applications don't integrate that well —something which starts right down at the file IO operations. This talk looks at how to make use of cloud object stores in Hadoop applications, including Hive and Spark. It will go from the foundational "what's an object store?" to the practical "what should I avoid" and the timely "what's new in Hadoop?" — the latter covering the improved S3 support in Hadoop 2.8+. I'll explore the details of benchmarking and improving object store IO in Hive and Spark, showing what developers can do in order to gain performance improvements in their own code —and equally, what they must avoid. Finally, I'll look at ongoing work, especially "S3Guard" and what its fast and consistent file metadata operations promise.

1) The document discusses how to create and improve open source software (OSS) projects and their communities. It addresses questions around the purpose of the OSS, languages used, versioning, and community engagement. 2) Key recommendations for building community include using English, being open to contributions, demonstrating stability and maintenance, and having a pluggable architecture. 3) The document debates tradeoffs like clean code vs quick contributions, focused vs feature-rich software, and localized vs global development and highlights the need to choose approaches given limitations. Overall it stresses continuous improvement over time.

This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.

This document summarizes Yurie Yamane's presentation on using mruby to make robots. It discusses using mruby and TOPPERS on the LEGO EV3 to create an inverted pendulum self-balancing robot. It then covers creating a DIY self-balancing robot using a Raspberry Pi, gyro sensor, DC motors, and a motor driver. Code examples are provided for reading sensor values, controlling motors with PWM, and implementing a balancing algorithm using a balancer class. Source code links are included for further reference.

It's Dangerous to GC alone. Take this! IBM's talk on work integrating the OMR GC into Ruby. OMR preview: goo.gl/P3yXuy

Presentation in RubyKaigi 2015

"Fireap" is a consul-based rapid propagative task runner for large systems. https://github.com/key-amb/fireap

Grifork runs defined tasks on the system in a way like tree's branching. Give grifork a list of hosts, then it creates a tree graph internally, and runs tasks in a top-down way.

- "poloxy" is a proxy system that pools and proxies alerts to prevent monitoring systems from bursting alerts to recipients. It works by having monitoring systems send alerts to "poloxy" instead of recipients. "poloxy" then enqueues alerts into a queue and has a worker dequeue alerts every minute and deliver them to original recipients, merging duplicate alerts and preventing repeated alerts from being delivered too frequently. This allows customizing how alerts are merged and controlling the frequency of alerts received by each recipient.

Tips and experiences about building and running Ruby on Solaris OS, with examples of bug fixes for supporting Solaris on Ruby.

Distributed Logging Architecture in Container Era The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.

Video presentation: https://www.youtube.com/watch?v=jLAFXQ1Av50 Most applications written in Ruby are great, but also exists evil code applying WOP techniques. There are many workarounds in several programming languages, but in Ruby, when it happens, the proportion is bigger. It's very easy to write Ruby code with collateral damage. You will see a collection of bad Ruby codes, with a description of how these codes affected negatively their applications and the solutions to fix and avoid them. Long classes, coupling, misapplication of OO, illegible code, tangled flows, naming issues and other things you can ever imagine are examples what you'll get.

This document summarizes a presentation about Meteor and React. It introduces Meteor as a full-stack platform for building web and mobile apps with JavaScript. It demonstrates storing messages in a Meteor collection and rendering them with React components. It also covers latency compensation, user accounts, and the future of Meteor integrating React, Redux, GraphQL and Socket.io on the backend.

成蹊大学「コース特殊講義Ａ （デザインとIT）」（坂井直樹先生）での井庭崇の講演スライド。2013年12月3日（火）。

The document discusses testing PHP applications using SimpleTest, Selenium IDE, and CakePHP. It provides an overview of these testing tools and frameworks and recommends them for testing PHP applications.

The document summarizes the new plugin API in Fluentd v0.14. Key points include: - The v0.12 plugin API was fragmented and difficult to write tests for. The v0.14 API provides a unified architecture. - The main plugin classes are Input, Filter, Output, Buffer, and plugins must subclass Fluent::Plugin::Base. - The Output plugin supports both buffered and non-buffered processing. Buffering can be configured by tags, time, or custom fields. - "Owned" plugins like Buffer are instantiated by primary plugins and can access owner resources. Storage is a new owned plugin for persistent storage. - New test drivers emulate plugin

The document discusses Fluentd's changes to its plugin API between versions 0.12 and 0.14. In 0.14, the API was overhauled to separate entry points from implementations and introduce a plugin base class to control data and control flow. A compatibility layer was added to allow most 0.12 plugins to work unmodified in 0.14 by handling calls to overridden methods. However, plugins that override certain methods like #emit may cause errors due to changes in how buffering works.

5/14~16で開催された スタートアップウィークエンド東京 Animeの「せいまち」チームのプレゼン資料です。 http://tokyo.startupweekend.org/

The document discusses testing practices for the Ruby programming language. It provides details on how to run various test suites that are part of the Ruby source code repository, including: 1. Running the "make test" command which runs sample tests, known bug tests, and tests defined in the test/ directory. 2. Running "make test-all" which runs core library and standard library tests under the test/ directory. 3. Running "make check" which builds encodings and extensions, runs all test tasks including test frameworks like Test::Unit and Minitest. 4. It also discusses strategies for merging test changes from external repositories like RubyGems and RDoc back into the Ruby source code

IBM is developing a just-in-time (JIT) compiler called Testarossa based on its Open Runtime (OMR) toolkit. The goal is to integrate the JIT compiler into MRI Ruby to improve performance without changing how MRI works. So far, the JIT supports most opcodes and can run Rails applications, but performance gains are modest. IBM hopes to collaborate with the Ruby community to further optimize Ruby and help make it faster.