Web program-peformance-optimization

This summary provides the key information from the document in 3 sentences: The document discusses using Lua and the ngx_lua module in UPYUN's CDN and API systems. It was built on top of Nginx with ngx_lua and contains over 40,000 lines of Lua code across many Lua modules. The system uses Lua for upstream health checking, load balancing, caching, and dynamic configuration of Nginx upstream servers.

The document describes the process of opening a TCP connection between a client and MySQL database, including the initial handshake and response packets. It then explains how the MRTE-Collector works by using message queues to capture and parse MySQL packets from the source database, and replay them to the target database using multiple SQL player threads. The MRTE-Collector publishes messages to RabbitMQ queues which routes the messages to the proper queues subscribed by MRTE-Player.

In this presentation, Chris Stivers, introduces the audience to Aerospike and provides tips on improving performance of Application written in Go. Tips include how to use memory more effectively in Go, and using Aerospike for high throughput / low latency transactions.

A monitoring system is arguably the most crucial system to have in place when administering and tweaking the performance of any database system. DBAs also find themselves with a variety of monitoring systems and plugins to use; ranging from small scripts in cron to complex data collection systems. In this talk, I’ll discuss how Box made a shift from the Cacti monitoring system and other various shell scripts to OpenTSDB and the changes made to our servers and daily interaction with monitoring to increase our agility in identifying and addressing changes in database behavior.

This document provides examples of common SQL anti-patterns and related NoSQL alternatives. It discusses issues like using tables as trees, caches, queues, or logs. It also addresses dynamic schema/table creation, stored procedures, row padding, complex joins, and object-relational mismatches. The document recommends alternatives like document databases, key-value stores, message brokers, and denormalization. It includes examples of modeling book data in MongoDB and Redis.

We recently replaced a proprietary API management solution with an in-house implementation built with nginx and Lua that let us get to a continuous delivery practice in a handful of months. Learn about our development process and the overall architecture that allowed us to write minimal amounts of code, enjoying native code performance while permitting interactive codeing, and how we leveraged other open source tools like Vagrant, Ansible, and OpenStack to build an automation-rich delivery pipeline. We will also take an in-depth look at our capacity management approach that differs from the rate limiting concept prevalent in the API community.

On Centralizing Logs with Syslog, LogStash, Elasticsearch, Kibana. Presentation from Radu Gheorghe from Sematext at Monitorama EU 2013.

Small Node.js proxy to turn a paginated JSON REST API into a CSV streaming download. Examples of code and patterns. Presented at the London Node User Group meetup, April 2014

The document summarizes Redis Modules and provides an overview of how to build and use Redis Modules. Some key points: - Redis Modules allow developers to extend Redis with new commands by dynamically loading libraries written in C/C++. This provides new functionality and integrates with existing Redis data types and commands. - The Modules API provides both high-level and low-level interfaces. The high-level API is similar to Lua but slower, while the low-level API exposes specific Redis commands and data types for better performance. - Building a simple module involves writing a command handler, validating arguments, making Redis calls, and returning a reply. Modules are initialized via an OnLoad function and new commands can

This document discusses building a distributed job queue in MongoDB. It begins by outlining the requirements of processing tasks across multiple servers. It then describes the design of storing messages and tasks in MongoDB collections, including different message and task states. It also covers techniques like composite tasks, reducing polling through tailable cursors, and optimizing performance through retry logic and oplog replay. The overall aim is to build an asynchronous and scalable way to process jobs in a distributed manner using MongoDB.

This document discusses using Redis and the Redis::Client Perl module to build scalable distributed job queues. It provides an overview of Redis, describing it as a key-value store that is simple, fast, and open-source. It then covers the various Redis data types like strings, lists, hashes, sets and sorted sets. Examples are given of how to work with these types using Redis::Client. The document discusses using Redis lists to implement job queues, with jobs added via RPUSH and popped via BLPOP. Benchmark results show the Redis-based job queue approach significantly outperforms using a MySQL jobs table with polling. Some caveats are provided about the benchmarks.

The document discusses using Lua modules with Nginx. It provides an overview of Nginx and Lua, describing their key features and popular uses. It then covers how to integrate Lua into Nginx using the ngx_lua module, allowing Lua code to run at various phases of the Nginx request lifecycle. Specific techniques demonstrated include using Lua for shared memory caching, logging, issuing subrequests, and network communication via cosockets. Examples of practical applications like load balancing and analytics tagging are also presented.

The document discusses using Node.js to build streaming services. It describes how Node.js allows for scalable server-side code using JavaScript and mentions libraries like JSONStream that can be used to parse JSON streams. The document also discusses different types of streaming like simplex, throughput, and duplex streaming and how to manage backpressure in streams.

This document describes how to set up monitoring for MySQL databases using Prometheus and Grafana. It includes instructions for installing and configuring Prometheus and Alertmanager on a monitoring server to scrape metrics from node_exporter and mysql_exporter. Ansible playbooks are provided to automatically install the exporters and configure Prometheus. Finally, steps are outlined for creating Grafana dashboards to visualize the metrics and monitor MySQL performance.

Presentation I gave during DevOps Days Warsaw 2014 about combining Elasticsearch, Logstash and Kibana together or use our Logsene solution instead of Elasticsearch.

This document discusses key metrics to monitor for Node.js applications, including event loop latency, garbage collection cycles and time, process memory usage, HTTP request and error rates, and correlating metrics across worker processes. It provides examples of metric thresholds and issues that could be detected, such as high garbage collection times indicating a problem or an event loop blocking issue leading to high latency.

An updated talk about how to use Solr for logs and other time-series data, like metrics and social media. In 2016, Solr, its ecosystem, and the operating systems it runs on have evolved quite a lot, so we can now show new techniques to scale and new knobs to tune. We'll start by looking at how to scale SolrCloud through a hybrid approach using a combination of time- and size-based indices, and also how to divide the cluster in tiers in order to handle the potentially spiky load in real-time. Then, we'll look at tuning individual nodes. We'll cover everything from commits, buffers, merge policies and doc values to OS settings like disk scheduler, SSD caching, and huge pages. Finally, we'll take a look at the pipeline of getting the logs to Solr and how to make it fast and reliable: where should buffers live, which protocols to use, where should the heavy processing be done (like parsing unstructured data), and which tools from the ecosystem can help.

This document provides an overview of Node.js and how to build web applications with it. It discusses asynchronous and synchronous reading and writing of files using the fs module. It also covers creating HTTP servers and clients to handle network requests, as well as using common Node modules like net, os, and path. The document demonstrates building a basic web server with Express to handle GET and POST requests, and routing requests to different handler functions based on the request path and method.

Node.js is an asynchronous JavaScript runtime that allows for efficient handling of I/O operations. The presentation discusses developing with Node.js by using modules from NPM, debugging with node-inspector, common pitfalls like blocking loops, and best practices like avoiding large heaps and offloading intensive tasks. Key Node.js modules demonstrated include Express for web frameworks and Socket.io for real-time applications.

This document discusses asynchronous I/O in Java and Scala using the Play Framework. It describes how LinkedIn uses a service-oriented architecture with hundreds of services making requests to each other. It then covers how Play supports non-blocking I/O using asynchronous code, promises, and futures to allow parallel requests without blocking threads. Key points covered include using map and flatMap to transform promises and futures, handling errors and timeouts, and the benefits of non-blocking I/O for scalability.

The document discusses Node.js and compares it to other technologies like CakePHP. It provides an overview of Node.js including its event-driven and asynchronous model, key features like the V8 engine and packages/modules, and frameworks like Express. It then demonstrates building a sample messaging application with a JSON API using both CakePHP and Node.js.

Node.js is a JavaScript runtime built on Chrome's V8 engine that allows building scalable network applications using JavaScript on the server-side. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, suitable for data-intensive real-time applications that run across distributed devices. Common uses of Node.js include building web servers, file upload clients, ad servers, chat servers, and any real-time data applications. The document provides an introduction to Node.js concepts like callbacks, blocking vs non-blocking code, the event loop, streams, events, and modules.

This document discusses REST (REpresentational State Transfer) and how to implement RESTful services on Android. It begins by defining REST and describing its core concepts like client-server architecture, statelessness, uniform interface, and CRUD (create, read, update, delete) operations. It then covers how to make HTTP requests in Android using libraries like HttpURLConnection and Apache HTTP Client. Helpful libraries for working with REST APIs are also presented, including Gson for JSON parsing and CRest for declarative REST clients. The document emphasizes best practices like performing HTTP calls in a background thread, persisting data to content providers, and minimizing network usage.

The document discusses the future of server-side JavaScript. It covers various Node.js frameworks and libraries that support both synchronous and asynchronous programming styles. CommonJS aims to provide interoperability across platforms by implementing synchronous proposals using fibers. Examples demonstrate how CommonJS allows for synchronous-like code while maintaining asynchronous behavior under the hood. Benchmarks show it has comparable performance to Node.js. The author advocates for toolkits over frameworks and continuing development of common standards and packages.