The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)Maarten Balliauw
The document discusses Azure Cache and Redis. It provides an overview of Redis, including its data types, transactions, pub/sub capabilities, scripting, and sharding/partitioning. It then discusses common patterns for using Redis, such as caching, counting likes on Facebook, getting the latest reviews, rate limiting, and autocompletion. The document emphasizes that Redis is very flexible and can be used for more than just caching, acting as a general datastore. It concludes by recommending a Redis reference book for further learning.
Sherlock Homepage - A detective story about running large web services - NDC ...Maarten Balliauw
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
We browse the Internet. We host our applications on a server or a cloud that is hooked up with a nice domain name. That’s all there is to know about DNS, right? This talk is a refresher about how DNS works. How we can use it and how it can affect availability of our applications. How we can use it as a means of configuring our application components. How this old geezer protocol is a resilient, distributed system that is used by every Internet user in the world. How we can use it for things that it wasn’t built for. Come join me on this journey through the innards of the web!
10 performance and scalability secrets of ASP.NET websitesoazabir
1) ASP.NET requires optimizations at the code, database, and configuration levels to scale to millions of hits out of the box. Common optimizations include tweaking process model settings, removing unnecessary pipeline components, and using compiled Linq queries.
2) Issues like application-level DOS attacks, slow profile provider stored procedures, and Linq to SQL performance problems can be addressed to improve scalability. Using a CDN can also help offload static content delivery.
3) Database queries must consider index usage and transaction isolation levels to prevent timeouts and deadlocks under high load.
The document discusses techniques for optimizing server performance, including reducing server load through HTTP caching, efficient content types, batching requests, and pushing content streams. It also discusses measuring performance metrics like response time and throughput, and tools for performance profiling like ASP.NET counters and load testing software. Specific techniques presented include using ETags, setting cache control headers, throttling requests, and choosing efficient data serialization formats and protocols.
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
Treasure Data is a data analytics service company that makes heavy use of Ruby in its platform and services. It uses Ruby for components like Fluentd (log collection), Embulk (data loading), scheduling, and its Rails-based API and console. Java and JRuby are also used for components involving Hadoop and Presto processing. The company's architecture includes collectors that ingest data, a PlazmaDB for storage, workers that process jobs on Hadoop and Presto clusters, and schedulers that queue and schedule those jobs using technologies like PerfectSched and PerfectQueue which are written in Ruby. Hive jobs are built programmatically using Ruby to generate configurations and submit the jobs to underlying Hadoop clusters.
The document describes Yahoo's failsafe mechanism for its homepage using Apache Storm and Apache Traffic Server. The key points are:
1. The failsafe architecture uses AWS components like EC2, ELB, S3 and autoscaling to serve traffic from failsafe servers if the primary servers fail.
2. Apache Traffic Server is used as a caching proxy between the user and origin servers. The "Escalate" plugin in ATS fetches content from failsafe servers if the origin server response is not good.
3. Apache Storm Crawler crawls content for different devices and maps URLs to the failsafe domain for storage in S3 with query parameters in the path. This provides more relevant fail
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)Maarten Balliauw
The document discusses Azure Cache and Redis. It provides an overview of Redis, including its data types, transactions, pub/sub capabilities, scripting, and sharding/partitioning. It then discusses common patterns for using Redis, such as caching, counting likes on Facebook, getting the latest reviews, rate limiting, and autocompletion. The document emphasizes that Redis is very flexible and can be used for more than just caching, acting as a general datastore. It concludes by recommending a Redis reference book for further learning.
Sherlock Homepage - A detective story about running large web services - NDC ...Maarten Balliauw
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
We browse the Internet. We host our applications on a server or a cloud that is hooked up with a nice domain name. That’s all there is to know about DNS, right? This talk is a refresher about how DNS works. How we can use it and how it can affect availability of our applications. How we can use it as a means of configuring our application components. How this old geezer protocol is a resilient, distributed system that is used by every Internet user in the world. How we can use it for things that it wasn’t built for. Come join me on this journey through the innards of the web!
10 performance and scalability secrets of ASP.NET websitesoazabir
1) ASP.NET requires optimizations at the code, database, and configuration levels to scale to millions of hits out of the box. Common optimizations include tweaking process model settings, removing unnecessary pipeline components, and using compiled Linq queries.
2) Issues like application-level DOS attacks, slow profile provider stored procedures, and Linq to SQL performance problems can be addressed to improve scalability. Using a CDN can also help offload static content delivery.
3) Database queries must consider index usage and transaction isolation levels to prevent timeouts and deadlocks under high load.
The document discusses techniques for optimizing server performance, including reducing server load through HTTP caching, efficient content types, batching requests, and pushing content streams. It also discusses measuring performance metrics like response time and throughput, and tools for performance profiling like ASP.NET counters and load testing software. Specific techniques presented include using ETags, setting cache control headers, throttling requests, and choosing efficient data serialization formats and protocols.
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
Treasure Data is a data analytics service company that makes heavy use of Ruby in its platform and services. It uses Ruby for components like Fluentd (log collection), Embulk (data loading), scheduling, and its Rails-based API and console. Java and JRuby are also used for components involving Hadoop and Presto processing. The company's architecture includes collectors that ingest data, a PlazmaDB for storage, workers that process jobs on Hadoop and Presto clusters, and schedulers that queue and schedule those jobs using technologies like PerfectSched and PerfectQueue which are written in Ruby. Hive jobs are built programmatically using Ruby to generate configurations and submit the jobs to underlying Hadoop clusters.
The document describes Yahoo's failsafe mechanism for its homepage using Apache Storm and Apache Traffic Server. The key points are:
1. The failsafe architecture uses AWS components like EC2, ELB, S3 and autoscaling to serve traffic from failsafe servers if the primary servers fail.
2. Apache Traffic Server is used as a caching proxy between the user and origin servers. The "Escalate" plugin in ATS fetches content from failsafe servers if the origin server response is not good.
3. Apache Storm Crawler crawls content for different devices and maps URLs to the failsafe domain for storage in S3 with query parameters in the path. This provides more relevant fail
This document summarizes a talk given at ApacheCon 2015 about replacing Squid with ATS (Apache Traffic Server) as the proxy server at Yahoo. It discusses the history of using Squid at Yahoo, limitations with Squid that prompted the switch to ATS, key differences in configuration between the two systems, examples of forwarding and reverse proxy use cases, and learnings around managing open source projects and migration testing.
A brief overview of what we do at Gruntwork. Learn what we mean by "DevOps as a Service" and how you can get your entire infrastructure, defined as code, in about a day. https://www.gruntwork.io/
Parse is a suite of cloud based APIs, services and libraries that focus on letting developers build out rich applications and less time dealing with the overhead of setting up and managing databases, push notifications, social sign on, analytics, and even hosting and servers.
In this series I'll overview the options around developing an application that leverages Parse, including using Cloud Code to deploy your Node.js app to Parse's own hosting service.
Firebase defines for everyone. It allows developers to store and sync data across applications through its realtime database and access it from anywhere through secure APIs. Firebase is a NoSQL database that uses JSON format for flexible data storage and scales horizontally across clusters. It is focused on mobile development and provides features like authentication, hosting, and analytics to help apps move fast without managing infrastructure.
The document discusses using Parse Cloud Code to build web applications, including basic operations like create, read, update, delete, how Parse and RESTful APIs work, and how to use Cloud Code to call external APIs, run background jobs, and include other JavaScript modules.
For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
This document provides an overview of Node.js, including what it is, its key features, and how to test a Node.js installation. Node.js is a JavaScript runtime environment that allows building scalable network applications in a single programming language. It uses non-blocking I/O and event-driven architecture, making it suitable for data-intensive real-time applications. The document demonstrates creating an HTTP server in Node.js, using the built-in URL module, and shows how Node.js uses an asynchronous and event-driven platform. It concludes by providing a link to a demo login application built with Node.js on GitHub.
Apache spark with akka couchbase code by bhawaniBhawani N Prasad
1) The document describes how to build a REST API using Spark, Spray, and Couchbase to enable fast CRUD operations on Couchbase data from Spark.
2) Key aspects include connecting Spark and Couchbase using the spark-connector library, starting a Spray server to host the REST API, and implementing CRUD routes using Couchbase APIs for RDDs like saveToCouchbase(), couchbaseGet(), couchbaseView(), and couchbaseQuery().
3) The REST API provides routes for inserting, updating, and getting documents from Couchbase via key-value, views, and N1QL operations on RDDs.
Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc.
Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.
(SDD402) Amazon ElastiCache Deep Dive | AWS re:Invent 2014Amazon Web Services
Peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns of our Memcached and Redis offerings and how customers have used them for in-memory operations and achieved improved latency and throughput for applications. During this session, we review best practices, design patterns, and anti-patterns related to Amazon ElastiCache.
Apache can function as both a forward and reverse proxy server. To configure it as a proxy, enable the proxy module, turn on proxy requests, and specify which clients can access the proxy. The proxy caches frequently accessed pages to improve performance and reduce bandwidth. It also provides security, access control, and logging of internet traffic on the network.
Modern tooling to assist with developing applications on FreeBSDSean Chittenden
Discuss a workflow and the tooling for FreeBSD engineers to develop locally on their laptop (OS-X, Windows, or FreeBSD), and push applications to bare metal or the cloud. The tooling required to provide good automation from a developer laptop to production takes time to evolve, however this lecture will jumpstart a series of best practices for FreeBSD engineers who want to see their business applications run on FreeBSD.
Full stack development with node and NoSQL - All Things Open - October 2017Matthew Groves
What is different about this generation of web applications? A solid development approach must consider latency, throughput, and interactivity demanded by users users across mobile devices, web browsers, and IoT. These applications often use NoSQL to support a flexible data model and easy scalability required for modern development.
A full stack application (composed of Couchbase, WebAPI, Angular2, and ASP.NET/ASP.NET Core) will be demonstrated in this session. The individual parts of a stack may vary, but the overall design is the focus.
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!MongoDB
Oh no! My backups aren't progressing! If something happens in production now, and I don't have current backups, I'll be out of a job for sure!
If these words resonate with you, don’t worry; you’re not the only one! Backup issues are one of the most common topics we deal with in Technical Services. In this talk, we will go through the backup flow, talk about where things might go wrong, and the symptoms you will see in the logs and the UI. We will also talk about other commands you can run to confirm the diagnosis, and how support can assist if you’re still stuck. Finally, we will talk about the new backup architecture in 4.2 and how it simplifies some of these concerns. This session is suitable for those with all levels of Ops Manager experience, but attendees should have a basic understanding of MongoDB’s replication process before attending this session.
After this talk, you will have leveled up your backup superpowers, and can swoop in to save your job (and the day)!
Caching has been an essential strategy for greater performance in computing since the beginning of the field. Nearly all applications have data access patterns that make caching an attractive technique, but caching also has hidden trade-offs related to concurrency, memory usage, and latency.
As we build larger distributed systems, caching continues to be a critical technique for building scalable, high-throughput, low-latency applications. Large systems tend to magnify the caching trade-offs and have created new approaches to distributed caching. There are unique challenges in testing systems like these as well.
Ehcache and Terracotta provide a unique way to start with simple caching for a small system and grow that system over time with a consistent API while maintaining low-latency, high-throughput caching.
The document discusses analyzing log data using Apache Spark. It covers challenges with log data like schema mediation and feature engineering to transform log records into vectors. It also discusses visualizing the structured data using dimensionality reduction techniques like PCA and self-organizing maps to find outliers. The document provides examples of analyzing log data to identify the most frequent error levels and applications generating logs.
- Overview of new features added to ERPNext including over production allowance percentage setting, tracking operations in production orders, and minor fixes.
- Updates on support work including migrating 5 accounts to V5 and being dedicated to fixing V5 issues.
- Details on the first paid feature for customer credit days and accounting reports being 20 times faster.
- Notes from website cleanup and fixes to issues with product search, cancel buttons, time logs, and dates in exports.
- Prototyping work done with iXsystems including a product configurator and margin calculation.
This document summarizes a talk given at ApacheCon 2015 about replacing Squid with ATS (Apache Traffic Server) as the proxy server at Yahoo. It discusses the history of using Squid at Yahoo, limitations with Squid that prompted the switch to ATS, key differences in configuration between the two systems, examples of forwarding and reverse proxy use cases, and learnings around managing open source projects and migration testing.
A brief overview of what we do at Gruntwork. Learn what we mean by "DevOps as a Service" and how you can get your entire infrastructure, defined as code, in about a day. https://www.gruntwork.io/
Parse is a suite of cloud based APIs, services and libraries that focus on letting developers build out rich applications and less time dealing with the overhead of setting up and managing databases, push notifications, social sign on, analytics, and even hosting and servers.
In this series I'll overview the options around developing an application that leverages Parse, including using Cloud Code to deploy your Node.js app to Parse's own hosting service.
Firebase defines for everyone. It allows developers to store and sync data across applications through its realtime database and access it from anywhere through secure APIs. Firebase is a NoSQL database that uses JSON format for flexible data storage and scales horizontally across clusters. It is focused on mobile development and provides features like authentication, hosting, and analytics to help apps move fast without managing infrastructure.
The document discusses using Parse Cloud Code to build web applications, including basic operations like create, read, update, delete, how Parse and RESTful APIs work, and how to use Cloud Code to call external APIs, run background jobs, and include other JavaScript modules.
For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
This document provides an overview of Node.js, including what it is, its key features, and how to test a Node.js installation. Node.js is a JavaScript runtime environment that allows building scalable network applications in a single programming language. It uses non-blocking I/O and event-driven architecture, making it suitable for data-intensive real-time applications. The document demonstrates creating an HTTP server in Node.js, using the built-in URL module, and shows how Node.js uses an asynchronous and event-driven platform. It concludes by providing a link to a demo login application built with Node.js on GitHub.
Apache spark with akka couchbase code by bhawaniBhawani N Prasad
1) The document describes how to build a REST API using Spark, Spray, and Couchbase to enable fast CRUD operations on Couchbase data from Spark.
2) Key aspects include connecting Spark and Couchbase using the spark-connector library, starting a Spray server to host the REST API, and implementing CRUD routes using Couchbase APIs for RDDs like saveToCouchbase(), couchbaseGet(), couchbaseView(), and couchbaseQuery().
3) The REST API provides routes for inserting, updating, and getting documents from Couchbase via key-value, views, and N1QL operations on RDDs.
Presented by Rafal Kuć, Consultant and Software engineer, , Sematext Group, Inc.
Even though Solr can run without causing any troubles for long periods of time it is very important to monitor and understand what is happening in your cluster. In this session you will learn how to use various tools to monitor how Solr is behaving at a high level, but also on Lucene, JVM, and operating system level. You'll see how to react to what you see and how to make changes to configuration, index structure and shards layout using Solr API. We will also discuss different performance metrics to which you ought to pay extra attention. Finally, you'll learn what to do when things go awry - we will share a few examples of troubleshooting and then dissect what was wrong and what had to be done to make things work again.
(SDD402) Amazon ElastiCache Deep Dive | AWS re:Invent 2014Amazon Web Services
Peek behind the scenes to learn about Amazon ElastiCache's design and architecture. See common design patterns of our Memcached and Redis offerings and how customers have used them for in-memory operations and achieved improved latency and throughput for applications. During this session, we review best practices, design patterns, and anti-patterns related to Amazon ElastiCache.
Apache can function as both a forward and reverse proxy server. To configure it as a proxy, enable the proxy module, turn on proxy requests, and specify which clients can access the proxy. The proxy caches frequently accessed pages to improve performance and reduce bandwidth. It also provides security, access control, and logging of internet traffic on the network.
Modern tooling to assist with developing applications on FreeBSDSean Chittenden
Discuss a workflow and the tooling for FreeBSD engineers to develop locally on their laptop (OS-X, Windows, or FreeBSD), and push applications to bare metal or the cloud. The tooling required to provide good automation from a developer laptop to production takes time to evolve, however this lecture will jumpstart a series of best practices for FreeBSD engineers who want to see their business applications run on FreeBSD.
Full stack development with node and NoSQL - All Things Open - October 2017Matthew Groves
What is different about this generation of web applications? A solid development approach must consider latency, throughput, and interactivity demanded by users users across mobile devices, web browsers, and IoT. These applications often use NoSQL to support a flexible data model and easy scalability required for modern development.
A full stack application (composed of Couchbase, WebAPI, Angular2, and ASP.NET/ASP.NET Core) will be demonstrated in this session. The individual parts of a stack may vary, but the overall design is the focus.
MongoDB World 2019: Becoming an Ops Manager Backup Superhero!MongoDB
Oh no! My backups aren't progressing! If something happens in production now, and I don't have current backups, I'll be out of a job for sure!
If these words resonate with you, don’t worry; you’re not the only one! Backup issues are one of the most common topics we deal with in Technical Services. In this talk, we will go through the backup flow, talk about where things might go wrong, and the symptoms you will see in the logs and the UI. We will also talk about other commands you can run to confirm the diagnosis, and how support can assist if you’re still stuck. Finally, we will talk about the new backup architecture in 4.2 and how it simplifies some of these concerns. This session is suitable for those with all levels of Ops Manager experience, but attendees should have a basic understanding of MongoDB’s replication process before attending this session.
After this talk, you will have leveled up your backup superpowers, and can swoop in to save your job (and the day)!
Caching has been an essential strategy for greater performance in computing since the beginning of the field. Nearly all applications have data access patterns that make caching an attractive technique, but caching also has hidden trade-offs related to concurrency, memory usage, and latency.
As we build larger distributed systems, caching continues to be a critical technique for building scalable, high-throughput, low-latency applications. Large systems tend to magnify the caching trade-offs and have created new approaches to distributed caching. There are unique challenges in testing systems like these as well.
Ehcache and Terracotta provide a unique way to start with simple caching for a small system and grow that system over time with a consistent API while maintaining low-latency, high-throughput caching.
The document discusses analyzing log data using Apache Spark. It covers challenges with log data like schema mediation and feature engineering to transform log records into vectors. It also discusses visualizing the structured data using dimensionality reduction techniques like PCA and self-organizing maps to find outliers. The document provides examples of analyzing log data to identify the most frequent error levels and applications generating logs.
- Overview of new features added to ERPNext including over production allowance percentage setting, tracking operations in production orders, and minor fixes.
- Updates on support work including migrating 5 accounts to V5 and being dedicated to fixing V5 issues.
- Details on the first paid feature for customer credit days and accounting reports being 20 times faster.
- Notes from website cleanup and fixes to issues with product search, cancel buttons, time logs, and dates in exports.
- Prototyping work done with iXsystems including a product configurator and margin calculation.
Carson City Library - Board of Trustee 5/26/2016: 3D PrintingSena Loyd
The document discusses establishing a policy for the Carson City Library's new 3D printers, including introducing patrons to innovative technologies, only allowing lawful uses, and providing examples of other library 3D printing policies which restrict prohibited content and set procedures for design submission and cost recovery.
A personagem feminina entra em um estado de perturbação emocional após sentir uma forte atração por um colega, experienciando uma mistura de sentimentos dentro de si que a deixam inquieta e confusa. Ela não compreende o que está sentindo e busca entender os processos internos que a afetam, comparando-os à mistura de elementos químicos.
SIMULATION OF PRESSURE VARIATIONS WITHIN KIMILILI WATER SUPPLY SYSTEM USING E...IAEME Publication
Water Supply system is a system of engineered hydrologic and hydraulic components which provide water supply for domestic use, industrial purposes, fire fighting and so on. The system
comprises of intake structures, treatment units, storage tanks and distribution systems. A well designed water supply system is meant to operate optimally such that consumers have access to portable water of sufficient pressure and quality at all times. However during operations of water supply systems, cases of pressure drops, Leakages and contamination occur and the main challenge is the lack of a simple tool to accurately predict zones of low pressures and areas where quality is compromised.
Randy Oostra – 2014 nominee for Modern Healthcare’s Community Leadership Award Modern Healthcare
Randy Oostra – 2014 nominee for Modern Healthcare’s Community Leadership Award.
The success of the healthcare industry depends on leaders who define themselves by leading efforts to change lives and contribute to their communities through their work. But many go above and beyond commitments central to their roles, reaching out to support causes that may be wholly unrelated to healthcare, but which build and sustain strong communities and the quality of life within them. Modern Healthcare's Community Leadership Awards was established to recognize these leaders while bringing attention to the worthy causes they support. Modern Healthcare's Community Leadership Awards was established to recognize these leaders while bringing attention to the worthy causes they support.
http://www.modernhealthcare.com/section/community-leadership
Execução Distribuída: O Jeito de Trabalhar das Empresas do FuturoRegy Andrade
O documento discute a execução distribuída como o futuro do trabalho, onde processos e tarefas são realizados por pessoas internas, externas e robôs de forma integrada. A execução distribuída permite automação, escalabilidade, redução de custos e foco no negócio principal através da fragmentação e distribuição inteligente de tarefas. O documento fornece etapas e pilares para implementar esse modelo, como automação de processos, gestão de informações, comunicação e conexões entre as partes.
Quotes for Entrepreneurs - by Saud Masud, CEO, Vector Partners (Pvt.) Ltd.Khawaja Saud Masud
10 quotes to inspire action. Compiled and shared by Saud Masud, CEO, Vector Partners (Pvt.) Ltd. We encourage you to internalize these quotes and implement them starting today.
Sonia is a new child in the author's program who has Spina Bifida and uses a wheelchair. She is bilingual in English and French. Her family is new to Toronto and is interested in sports and recreation programs. They asked the author for advice on services available in the area to help Sonia and children with special needs.
The document discusses user experience design for mobile applications and digital services. It covers topics such as designing for engagement, defining moments of truth, sequencing experiences, and extending core experiences across various touchpoints. Componentization, service blueprints, emotional journey maps are presented as tools to design experiences. The document also discusses designing software as a system and considering the entire user interface as the system. It provides examples of extending listening, photo, and iPhone experiences across different stages from awareness to sharing content. The goal is to create enjoyable, predictable, and content-focused experiences for users.
Congestion avoidance mechanisms aim to predict impending congestion and reduce data transmission rates before packet loss occurs. Three main methods are DEC bit, Random Early Detection (RED), and source-based approaches. DEC bit uses routers to explicitly notify sources of congestion. RED drops packets probabilistically based on average queue length to implicitly notify sources. Source-based methods monitor round-trip times and window sizes to detect congestion and adjust transmission rates accordingly.
Tutorial de como crear particiones de disco duro en windows 10luisberazaarieta
El documento proporciona un tutorial de 12 pasos para crear particiones de disco duro en Windows 10, comenzando con la apertura del administrador de discos, seleccionando un disco con espacio disponible para reducir su volumen, creando un nuevo volumen simple en el espacio liberado, asignando una letra de identificación y un nombre a la nueva partición, y finalizando el proceso.
The document defines and describes monsoon climate regions, which experience seasonal reversal of wind directions and the accompanying wet and dry periods. Monsoon climates are found in South and Southeast Asia, parts of Australia and Africa, and some areas of the southern United States. They are characterized by heavy rainfall from May to October brought by southwesterly winds, and generally dry, cool conditions from November to March under northeasterly winds. Vegetation in these regions includes deciduous forests and crops like rice, sugarcane, bananas, and tea.
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
Sherlock Homepage - A detective story about running large web services (VISUG...Maarten Balliauw
The site was slow. CPU and memory usage everywhere! Some dead objects in the corner. Something terrible must have happened! We have some IIS logs. Some traces from a witness. But not enough to find out what was wrong. In this session, we’ll see how effective telemetry, a profiler or two as well as a refresher of how IIS runs our ASP.NET web applications can help solve this server murder mystery.
Scaling asp.net websites to millions of usersoazabir
This document discusses various techniques for optimizing ASP.NET applications to scale from thousands to millions of users. It covers topics such as preventing denial of service attacks, optimizing the ASP.NET process model and pipeline, reducing the size of ASP.NET cookies on static content, improving System.net settings, optimizing queries to ASP.NET membership providers, issues with LINQ to SQL, using transaction isolation levels to prevent deadlocks, and employing a content delivery network. The overall message is that ASP.NET requires various "hacks" at the code, database, and configuration levels to scale to support millions of hits.
The document provides an overview of middleware in Node.js and Express. It defines middleware as functions that have access to the request and response objects and can run code and make changes to these objects before the next middleware in the chain. It discusses common uses of middleware like logging, authentication, parsing request bodies. It also covers Connect middleware and how Express builds on Connect by adding features like routing and views. Key aspects covered include the middleware pipeline concept, error handling with middleware, and common middleware modules.
(WEB301) Operational Web Log Analysis | AWS re:Invent 2014Amazon Web Services
Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.
Building Continuous Application with Structured Streaming and Real-Time Data ...Databricks
This document summarizes a presentation about building a structured streaming connector for continuous applications using Azure Event Hubs as the streaming data source. It discusses key design considerations like representing offsets, implementing the getOffset and getBatch methods required by structured streaming sources, and challenges with testing asynchronous behavior. It also outlines issues contributed back to the Apache Spark community around streaming checkpoints and recovery.
This document provides an overview of the MEAN stack and demonstrates how to build a sample application with it. It begins with defining each component of the MEAN stack: MongoDB as the database, Express as the web application framework, AngularJS for the frontend framework, and Node.js as the runtime environment. It then demonstrates setting up a basic Express app, integrating authentication with Passport, and interacting with MongoDB using Mongoose. The document also discusses key concepts like asynchronous I/O in Node.js and model-view-controller patterns in AngularJS. Overall, it serves as a high-level introduction to the technologies that make up the MEAN stack.
What is going on - Application diagnostics on Azure - TechDays FinlandMaarten Balliauw
We all like building and deploying cloud applications. But what happens once that’s done? How do we know if our application behaves like we expect it to behave? Of course, logging! But how do we get that data off of our machines? How do we sift through a bunch of seemingly meaningless diagnostics? In this session, we’ll look at how we can keep track of our Azure application using structured logging, AppInsights and AppInsights analytics to make all that data more meaningful.
How and why we evolved a legacy Java web application to Scala... and we are s...Katia Aresti
Applications get old, and technology moves fast. Overtime, adding or modifying functionalities might become as expensive as re-coding everything all from scratch. But rewriting a complete website and its functionalities it’s hard if we want to minimize the risks of breaking existing functionalities and specially when this application fits in a ecosystem and interacts with other pieces of software and teams.
In this session, you will learn how we moved from a legacy java monolithic website using scala PlayFramework, AngularJS, Elasticsearch and MongoDB, how we built a multi service and REST oriented architecture, which were the technical and human problems we encountered and how we managed to solved them.
A Progressive Web Application has a lot of advantages compared to a traditional website. A PWA is user-friendly, increases customer loyalty and conversions. Curious how PWA looks like in real-life?. Curious how PWA looks like in real-life? Check out Dutch’s first PWA (Vue Storefront + Magento 2) webshop, a scoop in e-commerce land! Vendic presents Meubelplaats.nl
A webshop in need of renovation, with a back-end that didn’t meet the requirements and demands of Meubelplaats. They trusted Vendic to develop their new webshop as a PWA. And behold! A smarter, faster website with the look & feel of an app, but with a powerful Magento 2 engine.
1. The document discusses various optimizations that can be made to an ASP.NET MVC application to improve performance, including compiled LINQ queries, URL caching, and data caching.
2. Benchmark results show that optimizing partial view rendering, LINQ queries, and URL generation improved performance from 8 requests/second to 61.5 requests/second. Additional caching of URLs, statistics, and content improved performance to over 400 requests/second.
3. Turning off ASP.NET debug mode also provided a significant performance boost, showing the importance of running production sites in release mode.
Murakumo is an open-source IaaS cloud controller and API orchestrator developed in 2012 to manage virtual machines, storage, and networks. It uses a thin controller and rich node agent architecture with asynchronous job queue processing. It supports Linux KVM and uses a simple design intended for easy operation and maintenance.
The document provides an overview of the Open Grid Computing Environments (OGCE) project, which develops and packages software for science gateways and resources. Key components discussed include the OGCE portal for building grid portals, Axis services for resource discovery and prediction, a workflow suite, and JavaScript and tag libraries. The document describes downloading and installing the OGCE software, which can be done with a single command, and discusses some of the portlets, services, and components included in the OGCE toolkit.
This document discusses Infrastructure as Code (IaC) principles and practices. It describes how a software company implemented IaC to automate their infrastructure provisioning and application deployments using Kubernetes and Google Cloud Platform. They used Ansible playbooks to define and build their infrastructure and container images. Kubernetes configuration files standardized across environments allow for fast, idempotent deployments. Continuous integration tests changes at each stage from unit testing to integration testing after deployments. This implementation of IaC provides many wins including easily reproducible, consistent systems and improved deployment processes.
Meteor is a reactive web application framework that uses JavaScript on both the client and server. It provides reactivity through Tracker.autorun, which re-runs functions automatically when reactive data sources change. Meteor uses DDP for client-server communication and Minimongo, a MongoDB implementation, for client-side data caching. The document provides steps for creating a basic Meteor application with user accounts, routing, schemas, forms, and template helpers to display posts data reactively.
The aim of this report is to introduce developers to the world of Magento optimization, giving suggestions and practical examples of the best practices to apply.
My Slides about creating web sites which could also be useable even if you are not online! From Web Storages to Service Workers.
Presented at Mobiletech Conference in Munich March 2017
Eagle from eBay at China Hadoop Summit 2015Hao Chen
The document summarizes Hadoop, a full-stack real-time monitoring framework for eBay's Hadoop clusters. It discusses eBay's large-scale Hadoop environment with over 10 clusters, 10,000 nodes, and 50,000 jobs/day. It then introduces Eagle, the uniform monitoring framework, which consists of the Eagle framework and Eagle apps. The framework provides scalable real-time monitoring capabilities and the apps provide domain-specific monitoring for Hadoop, Spark, HBase etc. It highlights two Eagle apps: JPA for job performance monitoring and DAM for security monitoring.
The Future is Now: Leveraging the Cloud with RubyRobert Dempsey
My presentation from the Ruby Hoedown on cloud computing and how Ruby developers can take advantage of cloud services to build scalable web applications.
Similar to Sherlock Homepage - A detective story about running large web services - WebNextConf 2015 (20)
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
The C# nullability features help you minimize the likelihood of encountering that dreaded System.NullReferenceException. Nullability syntax and annotations give hints as to whether a type can be nullable or not, and better static analysis is available to catch unhandled nulls while developing your code. What's not to like?
Introducing explicit nullability into an existing code bases is a Herculean effort. There's much more to it than just sprinkling some `?` and `!` throughout your code. It's not a silver bullet either: you'll still need to check non-nullable variables for null.
In this talk, we'll see some techniques and approaches that worked for me, and explore how you can migrate an existing code base to use the full potential of C# nullability.
Nerd sniping myself into a rabbit hole... Streaming online audio to a Sonos s...Maarten Balliauw
After buying a set of Sonos-compatible speakers at IKEA, I was disappointed there's no support for playing audio from a popular video streaming service. They stream Internet radio, podcasts and what not. Well, not that service I want it to play!
Determined - and not knowing how deep the rabbit hole would be - I ventured on a trip that included network sniffing on my access point, learning about UPnP and running a web server on my phone (without knowing how to write anything Android), learning how MP4 audio is packaged (and has to be re-packaged). This ultimately resulted in an Android app for personal use, which does what I initially wanted: play audio from that popular video streaming service on Sonos.
Join me for this story about an adventure that has no practical use, probably violates Terms of Service, but was fun to build!
Building a friendly .NET SDK to connect to SpaceMaarten Balliauw
Space is a team tool that integrates chats, meetings, git hosting, automation, and more. It has an HTTP API to integrate third party apps and workflows, but it's massive! And slightly opinionated.
In this session, we will see how we built the .NET SDK for Space, and how we make that massive API more digestible. We will see how we used code generation, and incrementally made the API feel more like a real .NET SDK.
Microservices for building an IDE - The innards of JetBrains Rider - NDC Oslo...Maarten Balliauw
Ever wondered how IDE’s are built? In this talk, we’ll skip the marketing bit and dive into the architecture and implementation of JetBrains Rider. We’ll look at how and why we have built (and open sourced) a reactive protocol, and how the IDE uses a “microservices” architecture to communicate with the debugger, Roslyn, a WPF renderer and even other tools like Unity3D. We’ll explore how things are wired together, both in-process and across those microservices.
Indexing and searching NuGet.org with Azure Functions and Search - .NET fwday...Maarten Balliauw
Which NuGet package was that type in again? In this session, let's build a "reverse package search" that helps finding the correct NuGet package based on a public type.
Together, we will create a highly-scalable serverless search engine using Azure Functions and Azure Search that performs 3 tasks: listening for new packages on NuGet.org (using a custom binding), indexing packages in a distributed way, and exposing an API that accepts queries and gives our clients the best result.
NDC Sydney 2019 - Microservices for building an IDE – The innards of JetBrain...Maarten Balliauw
Ever wondered how IDE’s are built? In this talk, we’ll skip the marketing bit and dive into the architecture and implementation of JetBrains Rider.
We’ll look at how and why we have built (and open sourced) a reactive protocol, and how the IDE uses a “microservices” architecture to communicate with the debugger, Roslyn, a WPF renderer and even other tools like Unity3D. We’ll explore how things are wired together, both in-process and across those microservices. Let’s geek out!
JetBrains Australia 2019 - Exploring .NET’s memory management – a trip down m...Maarten Balliauw
This document discusses .NET memory management and the garbage collector. It explains that the CLR manages memory in a heap and the garbage collector reclaims unused memory. It describes how objects are allocated in generations and discusses how to help the garbage collector perform better by reducing allocations, using value types when possible, and properly disposing of objects. The document also provides examples of hidden allocations and demonstrates tools for analyzing memory usage like ClrMD and dotMemory Unit.
.NET Conf 2019 - Indexing and searching NuGet.org with Azure Functions and Se...Maarten Balliauw
Which NuGet package was that type in again? In this session, let's build a "reverse package search" that helps finding the correct NuGet package based on a public type.
Together, we will create a highly-scalable serverless search engine using Azure Functions and Azure Search that performs 3 tasks: listening for new packages on NuGet.org (using a custom binding), indexing packages in a distributed way, and exposing an API that accepts queries and gives our clients the best result.
https://blog.maartenballiauw.be/post/2019/07/30/indexing-searching-nuget-with-azure-functions-and-search.html
CloudBurst 2019 - Indexing and searching NuGet.org with Azure Functions and S...Maarten Balliauw
Which NuGet package was that type in again? In this session, let's build a "reverse package search" that helps finding the correct NuGet package based on a public type.
Together, we will create a highly-scalable serverless search engine using Azure Functions and Azure Search that performs 3 tasks: listening for new packages on NuGet.org (using a custom binding), indexing packages in a distributed way, and exposing an API that accepts queries and gives our clients the best result.
NDC Oslo 2019 - Indexing and searching NuGet.org with Azure Functions and SearchMaarten Balliauw
Which NuGet package was that type in again? In this session, let's build a "reverse package search" that helps finding the correct NuGet package based on a public type.
Together, we will create a highly-scalable serverless search engine using Azure Functions and Azure Search that performs 3 tasks: listening for new packages on NuGet.org (using a custom binding), indexing packages in a distributed way, and exposing an API that accepts queries and gives our clients the best result.
Approaches for application request throttling - Cloud Developer Days PolandMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Indexing and searching NuGet.org with Azure Functions and Search - Cloud Deve...Maarten Balliauw
This document discusses indexing NuGet packages using Azure Functions and Azure Search to power search capabilities in ReSharper and Rider. It proposes using Functions triggered by changes to the NuGet.org catalog to download packages, index them using reflection metadata, and upload the results to an Azure Search index. Each step would be a separate function to allow independent scaling. The final system would watch the catalog, index new/updated packages, and provide APIs for searching packages by type or namespace.
Approaches for application request throttling - dotNetCologneMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
CodeStock - Exploring .NET memory management - a trip down memory laneMaarten Balliauw
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
ConFoo Montreal - Microservices for building an IDE - The innards of JetBrain...Maarten Balliauw
Ever wondered how IDE’s are built? In this talk, we’ll skip the marketing bit and dive into the architecture and implementation of JetBrains Rider. We’ll look at how and why we have built (and open sourced) a reactive protocol, and how the IDE uses a “microservices” architecture to communicate with the debugger, Roslyn, a WPF renderer and even other tools like Unity3D. We’ll explore how things are wired together, both in-process and across those microservices. Let’s geek out!
ConFoo Montreal - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Microservices for building an IDE – The innards of JetBrains Rider - TechDays...Maarten Balliauw
Ever wondered how IDE’s are built? In this talk, we’ll skip the marketing bit and dive into the architecture and implementation of JetBrains Rider. We’ll look at how and why we have built (and open sourced) a reactive protocol, and how the IDE uses a “microservices” architecture to communicate with the debugger, Roslyn, a WPF renderer and even other tools like Unity3D. We’ll explore how things are wired together, both in-process and across those microservices. Let’s geek out!
JetBrains Day Seoul - Exploring .NET’s memory management – a trip down memory...Maarten Balliauw
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
The .NET Garbage Collector (GC) is really cool. It helps providing our applications with virtually unlimited memory, so we can focus on writing code instead of manually freeing up memory. But how does .NET manage that memory? What are hidden allocations? Are strings evil? It still matters to understand when and where memory is allocated. In this talk, we’ll go over the base concepts of .NET memory management and explore how .NET helps us and how we can help .NET – making our apps better. Expect profiling, Intermediate Language (IL), ClrMD and more!
VISUG - Approaches for application request throttlingMaarten Balliauw
Speaking from experience building a SaaS: users are insane. If you are lucky, they use your service, but in reality, they probably abuse. Crazy usage patterns resulting in more requests than expected, request bursts when users come back to the office after the weekend, and more! These all pose a potential threat to the health of our web application and may impact other users or the service as a whole. Ideally, we can apply some filtering at the front door: limit the number of requests over a given timespan, limiting bandwidth, ...
In this talk, we’ll explore the simple yet complex realm of rate limiting. We’ll go over how to decide on which resources to limit, what the limits should be and where to enforce these limits – in our app, on the server, using a reverse proxy like Nginx or even an external service like CloudFlare or Azure API management. The takeaway? Know when and where to enforce rate limits so you can have both a happy application as well as happy customers.
Test Management as Chapter 5 of ISTQB Foundation. Topics covered are Test Organization, Test Planning and Estimation, Test Monitoring and Control, Test Execution Schedule, Test Strategy, Risk Management, Defect Management
Blockchain and Cyber Defense Strategies in new genre timesanupriti
Explore robust defense strategies at the intersection of blockchain technology and cybersecurity. This presentation delves into proactive measures and innovative approaches to safeguarding blockchain networks against evolving cyber threats. Discover how secure blockchain implementations can enhance resilience, protect data integrity, and ensure trust in digital transactions. Gain insights into cutting-edge security protocols and best practices essential for mitigating risks in the blockchain ecosystem.
The presentation will delve into the ASIMOV project, a novel initiative that leverages Retrieval-Augmented Generation (RAG) to provide precise, domain-specific assistance to telecommunications engineers and technicians. The session will focus on the unique capabilities of Milvus, the chosen vector database for the project, and its advantages over other vector databases.
Attending this session will give you a deeper understanding of the potential of RAG and Milvus DB in telecommunications engineering. You will learn how to address common challenges in the field and enhance the efficiency of their operations. The session will equip you with the knowledge to make informed decisions about the choice of vector databases, and how best to use them for your use-cases
Building an Agentic RAG locally with Ollama and MilvusZilliz
With the rise of Open-Source LLMs like Llama, Mistral, Gemma, and more, it has become apparent that LLMs might also be useful even when run locally. In this talk, we will see how to deploy an Agentic Retrieval Augmented Generation (RAG) setup using Ollama, with Milvus as the vector database on your laptop. That way, you can also avoid being Rate Limited by OpenAI like I have been in the past.
This slide deck is a deep dive the Salesforce latest release - Summer 24, by the famous Stephen Stanley. He has examined the release notes very carefully, and summarised them for the Wellington Salesforce user group, virtual meeting June 27 2024.
Dev Dives: Mining your data with AI-powered Continuous DiscoveryUiPathCommunity
Want to learn how AI and Continuous Discovery can uncover impactful automation opportunities? Watch this webinar to find out more about UiPath Discovery products!
Watch this session and:
👉 See the power of UiPath Discovery products, including Process Mining, Task Mining, Communications Mining, and Automation Hub
👉 Watch the demo of how to leverage system data, desktop data, or unstructured communications data to gain deeper understanding of existing processes
👉 Learn how you can benefit from each of the discovery products as an Automation Developer
🗣 Speakers:
Jyoti Raghav, Principal Technical Enablement Engineer @UiPath
Anja le Clercq, Principal Technical Enablement Engineer @UiPath
⏩ Register for our upcoming Dev Dives July session: Boosting Tester Productivity with Coded Automation and Autopilot™
👉 Link: https://bit.ly/Dev_Dives_July
This session was streamed live on June 27, 2024.
Check out all our upcoming Dev Dives 2024 sessions at:
🚩 https://bit.ly/Dev_Dives_2024
The document discusses testing throughout the software development life cycle. It describes different software development models including sequential, incremental, and iterative models. It also covers different test levels from component and integration testing to system and acceptance testing. The document discusses different types of testing including functional and non-functional testing. It also covers topics like maintenance testing and triggers for additional testing when changes are made. Also covers concepts of Agile including DevOps, Shift Left Approach, TDD, BDD, ATDD, Retrospective and Process Improvement
How to Improve Your Ability to Solve Complex Performance ProblemsScyllaDB
This talk is really about problem solving. It’s about how we think about problems and how we resolve those problems in a deeply technical context. The main goal of the talk is the relay the lessons learned from a couple of decades working with and observing some of the best performance troubleshooters in the world.
The talk will be broken into 3 main parts.
1. Explain the basic process we must go through to solve a complex performance problem
2. Discuss some of the main factors that can inhibit our efforts
3. Discuss some of the techniques we can apply to improve our chances, including an almost fool proof method to reach a successful outcome
Specific technical examples from large enterprise customers using relational databases (Oracle primarily) will be used to illustrate the concepts.
Details of description part II: Describing images in practice - Tech Forum 2024BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and transcript: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Transcript: Details of description part II: Describing images in practice - T...BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and slides: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Metadata Lakes for Next-Gen AI/ML - DatastratoZilliz
As data catalogs evolve to meet the growing and new demands of high-velocity, unstructured data, we see them taking a new shape as an emergent and flexible way to activate metadata for multiple uses. This talk discusses modern uses of metadata at the infrastructure level for AI-enablement in RAG pipelines in response to the new demands of the ecosystem. We will also discuss Apache (incubating) Gravitino and its open source-first approach to data cataloging across multi-cloud and geo-distributed architectures.
Multimodal Retrieval Augmented Generation (RAG) with MilvusZilliz
We've seen an influx of powerful multimodal capabilities in many LLMs. In this talk, we'll vectorize a dataset of images and texts into the same embedding space, store them in Milvus, retrieve all relevant data using multilingual texts and/or images and input multimodal data as context into GPT-4o.
AI_dev Europe 2024 - From OpenAI to Opensource AIRaphaël Semeteys
Navigating Between Commercial Ownership and Collaborative Openness
This presentation explores the evolution of generative AI, highlighting the trajectories of various models such as GPT-4, and examining the dynamics between commercial interests and the ethics of open collaboration. We offer an in-depth analysis of the levels of openness of different language models, assessing various components and aspects, and exploring how the (de)centralization of computing power and technology could shape the future of AI research and development. Additionally, we explore concrete examples like LLaMA and its descendants, as well as other open and collaborative projects, which illustrate the diversity and creativity in the field, while navigating the complex waters of intellectual property and licensing.
Quality Patents: Patents That Stand the Test of TimeAurora Consulting
Is your patent a vanity piece of paper for your office wall? Or is it a reliable, defendable, assertable, property right? The difference is often quality.
Is your patent simply a transactional cost and a large pile of legal bills for your startup? Or is it a leverageable asset worthy of attracting precious investment dollars, worth its cost in multiples of valuation? The difference is often quality.
Is your patent application only good enough to get through the examination process? Or has it been crafted to stand the tests of time and varied audiences if you later need to assert that document against an infringer, find yourself litigating with it in an Article 3 Court at the hands of a judge and jury, God forbid, end up having to defend its validity at the PTAB, or even needing to use it to block pirated imports at the International Trade Commission? The difference is often quality.
Quality will be our focus for a good chunk of the remainder of this season. What goes into a quality patent, and where possible, how do you get it without breaking the bank?
** Episode Overview **
In this first episode of our quality series, Kristen Hansen and the panel discuss:
⦿ What do we mean when we say patent quality?
⦿ Why is patent quality important?
⦿ How to balance quality and budget
⦿ The importance of searching, continuations, and draftsperson domain expertise
⦿ Very practical tips, tricks, examples, and Kristen’s Musts for drafting quality applications
https://www.aurorapatents.com/patently-strategic-podcast.html
Leveraging AI for Software Developer Productivity.pptxpetabridge
Supercharge your software development productivity with our latest webinar! Discover the powerful capabilities of AI tools like GitHub Copilot and ChatGPT 4.X. We'll show you how these tools can automate tedious tasks, generate complete syntax, and enhance code documentation and debugging.
In this talk, you'll learn how to:
- Efficiently create GitHub Actions scripts
- Convert shell scripts
- Develop Roslyn Analyzers
- Visualize code with Mermaid diagrams
And these are just a few examples from a vast universe of possibilities!
Packed with practical examples and demos, this presentation offers invaluable insights into optimizing your development process. Don't miss the opportunity to improve your coding efficiency and productivity with AI-driven solutions.
MYIR Product Brochure - A Global Provider of Embedded SOMs & SolutionsLinda Zhang
This brochure gives introduction of MYIR Electronics company and MYIR's products and services.
MYIR Electronics Limited (MYIR for short), established in 2011, is a global provider of embedded System-On-Modules (SOMs) and
comprehensive solutions based on various architectures such as ARM, FPGA, RISC-V, and AI. We cater to customers' needs for large-scale production, offering customized design, industry-specific application solutions, and one-stop OEM services.
MYIR, recognized as a national high-tech enterprise, is also listed among the "Specialized
and Special new" Enterprises in Shenzhen, China. Our core belief is that "Our success stems from our customers' success" and embraces the philosophy
of "Make Your Idea Real, then My Idea Realizing!"
MYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
Sherlock Homepage - A detective story about running large web services - WebNextConf 2015
1. Sherlock Homepage
A detective story about running
large web services.
Maarten Balliauw
@maartenballiauw
(Optimizing web apps using AppInsights, memory and performance profiling)
4. Site unavailable!
Primary website location unavailable!
No problem: traffic manager in front – pfew!
Secondary location unavailable!
NuGet.org down…
Initial investigation & monitoring showed:
Primary & secondary website location instances all up
Machines available individually
Not through traffic manager and Azure load balancer
6. How did we find the issue?
Quote from “Mind Hunter” (written by an FBI profiler):
You have to be able to re-create the crime scene in your head. You need
to know as much as you can about the victim so that you can imagine
how she might have reacted. You have to be able to put yourself in her
place as the attacker threatens her with a gun or a knife, a rock, his fists,
or whatever.
You have to be able to feel her fear as he approaches her. You have to be
able to feel her pain. You have to try to imagine what she was going
through when he tortured her. You have to understand what it’s like to
scream in terror and agony, realizing that it won’t help, that it won’t get
him to stop. You have to know what it was like.
http://highscalability.com/blog/2015/7/30/how-debugging-is-like-hunting-serial-killers.html
7. How did we find the issue?
Debugging requires a particular sympathy for the machine. You must be able to run
the machine and networks of machines in your mind while simulating what-ifs based
on mere wisps of insight.
In reality: it was a hunch
Knowing the system you are working on – even by similarity
Empathy for what is going on in that system
Prior experience / insights
9. Who am I?
Maarten Balliauw
Antwerp, Belgium
Software Engineer, Microsoft
Founder, MyGet
AZUG
Focus on web
ASP.NET MVC, Azure, SignalR, ...
Former MVP Azure & ASPInsider
Big passion: Azure
http://blog.maartenballiauw.be
@maartenballiauw
Shameless self promotion: Pro NuGet - http://amzn.to/pronuget2
11. A bit of history
NuGet.org serves dependencies for .NET developers worldwide
On average good for ~8.000.000 to ~10.000.000 request per day
(333.333/hr or 5.555/min or 92/sec)
Built 4 years ago on top of a SQL database and OData services
Monolithic – site + NuGet feed are the same app
Minor improvements over the years
Some rough times Q2 2015
12. Architecture overview (Q2 2015)
Front end servers (2 regions)
NuGet.org
MVC + WCF OData
Search Service
Lucene based Web API
Search Service
(secondary region)
Lucene based Web API
Azure Storage
Lucene index
Download counts
Azure SQL Database
Packages metadata
Download counts
Jobs VMs
Create index from database
Create stats
Create download count reports
14. One of the services caused this…
SQL database?
Storage?
Search?
Metrics service?
return new HttpStatusCodeWithBodyResult(AvailabilityStatusCode(galleryServiceAvailable),
String.Format(CultureInfo.InvariantCulture,
StatusMessageFormat,
AvailabilityMessage(galleryServiceAvailable),
AvailabilityMessage(sqlAzureAvailable),
AvailabilityMessage(storageAvailable),
AvailabilityMessage(searchServiceAvailable),
AvailabilityMessage(metricsServiceAvailable),
HostMachine.Name));
15. Log spelunking
Check SQL database logs – we found we had none (fixed now)
Storage – storage statistics seemed stable
Search – no real pointers to issues there
Metrics service – very lightweight and has been stable for months
Start looking around at the crime scene!
IIS logs, event viewer on web servers
Profiling on web servers
19. Turns out it was search!
Profiling the search service revealed some things!
SearcherManager.cs checks Lucene index freshness on Get() – MaybeReopen()
StartReopen() blocks access to the index until finished
Part of HTTP request pipeline – blocking request handling
This was fixed by getting these calls out of the HTTP request path.
The suspect no longer had code available – used our informants
www.jetbrains.com/dotpeek
20. Search had some other flaws…
Actually also visible in the dotTrace snapshot we just saw: high GC!
Memory profiling the search service revealed some things! (I lost the actual traces )
21. Search had some other flaws…
High memory traffic on reading download count # for search ranking
The source: DownloadLookup.cs#L18
Fixed by:
Reusing objects (instead of new)
JsonStreamReader instead of JObject.Parse(theWorld)
22. In the meanwhile…
Added additional monitoring
Added additional tracing
Started looking into using AppInsights for better insights into application behavior
Events happening on the website
Requests
Exceptions
Execution and dependency times (basic but continuous profiling)
24. What we were seeing…
On the V2-based feeds:
500 Internal Server Error
during package restore
Response time goes up
while # of requests goes down
EventVwr on servers: lots of IIS crashes
Lots of crash dumps on web servers
25. So IIS crashes… Could it be?
HTTP.SYS tells us when things go wrong
D:WindowsSystem32LogFilesHTTPERR
2015-07-31 01:46:34 - 60810 - 80 HTTP/1.1 GET /api/v2/FindPackagesById()?id='...' - 1273337584
Connection_Abandoned_By_ReqQueue 86cd3cb1-729c-425c-898f-b15b0330bc38
Connection_Abandoned_By_ReqQueue
“A worker process from the application pool has quit unexpectedly or orphaned a pending
request by closing its handle. Specific to Windows Vista and Windows Server 2008.”
Gift from the gods: crash dumps!
C:ResourcesDirectory31edcaa5186f…...DiagnosticStoreWAD0104CrashDumps
27. An Exception crashes IIS?
Time to crank up the search engine queries!
Found a similar issue: unobserved task exceptions causing IIS to crash
// If metrics service is specified we post the data to it asynchronously.
if (_config != null && _config.MetricsServiceUri != null)
{
// Disable warning about not awaiting async calls
// because we are _intentionally_ not awaiting this.
#pragma warning disable 4014
Task.Run(() => PostDownloadStatistics(id, version, …));
#pragma warning restore 4014
}
28. TaskScheduler.UnobservedTaskException += (object sender, UnobservedTaskExceptionEventArgs excArgs) =>
{
// ... log it ...
excArgs.SetObserved();
};
Tasks and fire-and-forget are evil!
Unobserved task can cause the entire process to give up on Exception
Handle unobserved task Exceptions!
31. What we were seeing…
High response times on the servers
Resulting in higher than normal CPU usage on the servers
Azure would often auto-scale additional instances
Profiling the web application for NuGet.org still showed wait times with no obvious
cause…
33. Research
Reading and searching on what could be the cause of these issues
http://stackoverflow.com/questions/12304691/why-are-iis-threads-so-precious-as-compared-to-regular-clr-threads
http://www.monitis.com/blog/2012/06/11/improving-asp-net-performance-part3-threading/
https://msdn.microsoft.com/en-us/library/ms998549.aspx
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
https://support.microsoft.com/en-us/kb/821268
Consider minIoThreads and minWorkerThreads for Burst Load
If your application experiences burst loads where there are prolonged periods of inactivity between the burst
loads, the thread pool may not have enough time to reach the optimal level of threads. A burst load occurs
when a large number of users connect to your application suddenly and at the same time. The
minIoThreads and minWorkerThreads settings enable you to configure a minimum number of worker
threads and I/O threads for load conditions.
35. Making it permanent
Part of the NuGet Gallery startup script
# Increase the number of available IIS threads for high performance applications
# Uses the recommended values from http://msdn.microsoft.com/en-us/library/ms998549.aspx#scalenetchapt06_topic8
# Assumes running on two cores (medium instance on Azure)
&$appcmd set config /commit:MACHINE -section:processModel -maxWorkerThreads:100
&$appcmd set config /commit:MACHINE -section:processModel -minWorkerThreads:50
&$appcmd set config /commit:MACHINE -section:processModel -minIoThreads:50
&$appcmd set config /commit:MACHINE -section:processModel -maxIoThreads:100
# Adjust the maximum number of connections per core for all IP addresses
&$appcmd set config /commit:MACHINE -section:connectionManagement /+["address='*',maxconnection='240'"]
37. What we were seeing…
On the V2-based feeds:
Package restore timeouts coming from the WCF OData service
Occurs every 7-15 hours, fixes itself ~15 minutes later
Extreme load times on Get(Id=,Version=) – probably the cause of these timeouts
38. No easy way to reproduce…
Happening only on production
Observation after RDP-ing in: 100% CPU when it happens
No way to profile continuously – AppInsights did show us the entry point
Donut time again
The thing we recently changed was minIOThreads and throughput
The slow code path is FindPackagesById()
Makes HTTP calls to search service
What could this setting and HTTP calls have in common…
39. HttpClient, Async and multithreading
Interesting article benchmarking HttpClient in async and multithreading scenarios
Async + HttpClient are not limited in terms of concurrency
Many CPU’s and threads? Many HttpClients and requests
Many HttpClients and requests? Many TCP ports used on machine
Many TCP ports used? TCP port depletion
But aren’t ports reclaimed?
240 seconds TIME_WAIT (4 minutes)
Users also use up TCP ports
“As far as HTTP requests are concerned, a limit should always be set to
ServicePointManager.DefaultConnectionLimit. The limit should be large enough to allow a
good level of parallelism, but low enough to prevent performance and reliability problems (from the
exhaustion of ephemeral ports). “
40. Limiting HttpClient async concurrency
Set ServicePointManager properties on startup
Nagling – “bundle traffic in properly stuffed TCP packets”
Expect100Continue – “only send out traffic if server says 100 Continue”
Both optimizations also disabled
// Tune ServicePointManager
// (based on http://social.technet.microsoft.com/Forums/en-US/windowsazuredata/thread/d84ba34b-b0e0-
4961-a167-bbe7618beb83 and https://msdn.microsoft.com/en-
us/library/system.net.servicepointmanager.aspx)
ServicePointManager.DefaultConnectionLimit = 500;
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
43. What we were seeing…
Massive memory usage! Even when changing VM sizes.
100% of memory on a Medium Azure instance
100% of memory on a Large Azure instance
100% of memory on a X-Large Azure instance
44. What is eating this memory?
Memory profiling!
On the server?
Try to reproduce it?
Decided on the latter
46. .NET Memory Management 101
Memory Allocation
.NET runtime reserves region of address space for every new process
managed heap
Objects are allocated in the heap
Allocating memory is fast, it’s just adding a pointer
Some unmanaged memory is also consumed (not GC-ed)
.NET CLR, Dynamic libraries, Graphics buffer, …
Memory Release or “Garbage Collection” (GC)
Generations
Large Object Heap
47. .NET Memory Management 101
Memory Allocation
Memory Release or “Garbage Collection” (GC)
GC releases objects no longer in use by examining application roots
GC builds a graph that contains all the objects that are reachable from these
roots.
Object unreachable? GC removes the object from the heap, releasing memory
After the object is removed, GC compacts reachable objects in memory.
Generations
Large Object Heap
48. .NET Memory Management 101
Memory Allocation
Memory Release or “Garbage Collection” (GC)
Generations
Managed heap divided in segments: generation 0, 1 and 2
New objects go into Gen 0
Gen 0 full? Perform GC and promote all reachable objects to Gen 1. This is typically pretty fast.
Gen 1 full? Perform GC on Gen 1 and Gen 0. Promote all reachable objects to Gen 2.
Gen 2 full? Perform full GC (2, 1, 0). If not enough memory for new allocations, throws
OutOfMemoryException
Full GC has performance impact since all objects in managed heap are verified.
Large Object Heap
49. .NET Memory Management 101
Memory Allocation
Memory Release or “Garbage Collection” (GC)
Generations
Large Object Heap
Generation 0 Generation 1 Generation 2
Short-lived objects (e.g. Local
variables)
In-between objects Long-lived objects (e.g. App’s
main form)
50. .NET Memory Management 101
Memory Allocation
Memory Release or “Garbage Collection” (GC)
Generations
Large Object Heap
Large objects (>85KB) stored in separate segment of managed heap: Large
Object Heap (LOH)
Objects in LOH collected only during full garbage collection
Survived objects in LOH are not compacted (by default). This means that LOH
becomes fragmented over time.
Fragmentation can cause OutOfMemoryException
51. The .NET garbage collector
Simulates “infinite memory” by removing objects no longer needed
When does it run? Vague… But usually:
Out of memory condition – when the system fails to allocate or re-allocate memory
After some significant allocation – if X memory is allocated since previous GC
Failure of allocating some native resources – internal to .NET
Profiler – when triggered from profiler API
Forced – when calling methods on System.GC
Application moves to background
GC is not guaranteed to run
http://blogs.msdn.com/b/oldnewthing/archive/2010/08/09/10047586.aspx
http://blogs.msdn.com/b/abhinaba/archive/2008/04/29/when-does-the-net-compact-framework-garbage-collector-run.aspx
53. So our DI container? NInject?
Our memory profiling confirms it. The retained EntitiesContext also
retains entities and SQL connections.
Spelunking the NInject source code, we found the
GarbageCollectionCachePruner responsible for releasing objects.
Runs every 30 seconds (timer)
Releases objects only if GC happened in that time
GC is not guaranteed to run, so NInject potentially never releases objects
Known, old bug.
https://groups.google.com/forum/#!topic/ninject/PQNMIsQhCvE
http://stackoverflow.com/questions/16775362/ninject-caching-object-that-should-be-disposed-memoryleak
54. Replacing our DI container (Autofac)
Perform replacement
Run same analysis on new codebase
and verify objects are freed
Once deployed:
Immediate drop in response times
Memory usage now stable at ~4 GB
56. Conclusion
Debugging requires a particular sympathy for the machine. You must be
able to run the machine and networks of machines in your mind while
simulating what-ifs based on mere wisps of insight.
Bugs hide. They blend in. They can pass for "normal" which makes them tough to find.
One bug off the streets doesn’t mean all of them are gone. Sometimes one gone exposes
another.
Know your system, know your tools, know your options. Look for evidence.
Profilers (performance and memory), dump files, AppInsights and others
Dive in. It builds experience and makes solving the next crime scene easier.
https://msdn.microsoft.com/en-us/library/ee817663.aspx
One of these services was failing – resulting in status 500 – resulting in LB removing the instance from pool – all were removed from pool…
Serial killers are like bugs in the societal machine. They hide. They blend in. They can pass for "normal" which makes them tough to find. They attack weakness causing untold damage until caught. And they will keep causing damage until caught. They are always hunting for opportunity.
Yes it’s fun to work on new things. But try figuring out this stuff. Builds experience.
Profiling the search service issue - website side
* Open nuget-prod-0-v2gallery - 20150507 - 1215.dtt
* Notice lots of wait time (enable)
* Disable system methods - notice search hijacker is in there
* Overall, CPU looks healthy and further analysis doesn't really show us anything worth noticing
Profiling the search service issue - search service issue
* Open capture1.dtt
* Notice lots of lock contention
* Disable system methods - notice SearcherManager.Get() is top suspect
* Unfortunately, this code was deployed without symbols and we no longer had the sources around
* Decided to decompile using dotPeek and walk through code - which revealed something (next slide)
Profiling the search service issue - search service issue
* Open capture1.dtt
* Notice lots of lock contention
* Disable system methods - notice SearcherManager.Get() is top suspect
* Unfortunately, this code was deployed without symbols and we no longer had the sources around
* Decided to decompile using dotPeek and walk through code - which revealed something (next slide)
Perhaps do a quick tour of AppInsights depending on time left.
Analyzing a crash dump
* Open w3wp.exe.3344.dmp - explain this can be done with hardcore debugging tools but VS2015 works just as fine
* Explain dump summary - what we can see (threads, modules, last state when it all broke down)
* Becomes more interesting when we debug it - "Debug with managed only"
* Unhandled Exception, it says! That's probably the cause for IIS erroring out. But where does it come from...
* We can try the various windows for inspecting modules, threads, call stacks, ... but not al ot to see in there.
* We need DEBUGGER SYMBOLS! Unfortunately in this case the gallery was deployed without symbols. No symbols on teh build server either, for this deployment.
* DotPeek! Load the actual DLL, enable symbol server, start debugging again.
* See the exception now shows us the (decompiled) code. Not the real code but it does give us an idea.
* Unhandled exception in a task. Which crashes out IIS.
Reproducing production traffic
* We'll use jMeter - a tool that is perfect for replaying web server logs against other servers
* Explain IIS logs come from production
* Explain rconvlog tool to translate them to NCSA format
* Open up jMeter and explain how it is all linked to each other
Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
Application roots: Typically, these are global and static object pointers, local variables, and CPU registers.
Analyzing memory usage
* Open the Workspace [2015-08-05] [13-24].dmw workspace
* From the snapshots overview we already see a few interesting things: memory keeps going up, when GC'ed (using the profiler API) memory stays around
* Let's open one of the snapshots - snapshot #6 - why? As it says objects have been collected so we want to see what remained in memory
* From the overview: lots of objects on gen2 - eaning they survived many collections - dive in
* Seems NInject.Activation.Caching.Cache is keeping a lot of bytes in memory - dive in
* Lots of objects seem to be cached by NInjet. Check outgoing references to see what they are.
* The first ones look normal, e.g. the controller activator and so on is needed the entire time by ASP.NET MVC
* At [2] we see an EntitiesContext retained. That's weird: it should be disposed of after each request. Let's see if there are more of these still in memory.
* Snapshot - largest size - search entitiescontext - A LOT! Also in size relative to the snapshot size - dive in
* We can see 991 instances are kept around - who's holding on to them?
* Group by similar retention shows NInject for 596 of them. Wow.