A Tale of 2 Systems

Yahoo recently open-sourced Pulsar,  a highly scalable, low latency pub-sub messaging system running on commodity hardware. It provides simple pub-sub messaging semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. Pulsar is used across various Yahoo applications for large scale data pipelines. Learn more about Pulsar architecture and use-cases in this talk. Speakers: Matteo Merli from Pulsar team at Yahoo 

This document summarizes Kafka internals including how Zookeeper is used for coordination, how brokers store partitions and messages, how producers and consumers interact with brokers, how to ensure data integrity, and new features in Kafka 0.9 like security enhancements and the new consumer API. It also provides an overview of operating Kafka clusters including adding and removing brokers through reassignment.

This is the version of my Apache Performance Tuning deck I last presented at ApacheCon, in 2008 in Amsterdam.

DevOps and Continuous Delivery slides with Fabric8 (http://fabric8.io), HawtIO (http://hawt.io), Camel, ActiveMQ, Docker Jenkins, etc.

This document discusses Apache Flume and using its HBase sink. It provides an overview of Flume's architecture, components, and sources. It describes how the HBase sink works, its configuration options, and provides an example configuration for collecting data from a sequential source and storing it in an HBase table.

Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop. It's also enabling many real-time system frameworks and use cases. Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API. Also talk about the best practices involved in running a producer/consumer. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects. We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing Kafka ACLs and monitoring Consumer offsets.

OpenStack is an open source cloud computing platform that can manage large networks of virtual machines and physical servers. It uses a distributed architecture with components like Nova (compute), Swift (object storage), Cinder (block storage), and Quantum (networking). OpenStack has been successful due to its scalability, support for multiple hypervisors including Hyper-V, and compatibility with popular programming languages like Python. While OpenStack is best suited for large public and private clouds, its complex installation and lack of unified deployment tools can present challenges, especially for small to mid-sized clouds.

The document discusses converting a monolithic Node.js application into microservices and deploying them using Docker. It begins by defining microservices and their benefits. It then describes converting a sample pizza ordering application into independent microservices for handling messages, serving the frontend, and providing an API. Next, it covers patterns for networking microservices, including using an API gateway. It concludes by demonstrating how to deploy the microservices to Docker containers and use an orchestration tool like Kubernetes to manage them.

This document discusses containerization and the Docker ecosystem. It begins by describing the challenges of managing different software stacks across multiple environments. It then introduces Docker as a solution that packages applications into standardized units called containers that are portable and can run anywhere. The rest of the document covers key aspects of the Docker ecosystem like orchestration tools like Kubernetes and Docker Swarm, networking solutions like Flannel and Weave, storage solutions, and security considerations. It aims to provide an overview of the container landscape and components.

This slide show how can improve ceph storage performance using ssd and how to use ceph storage for container.

RavenDB is a schema-less document database that offers fully ACID transactions, fast and flexible search, replication, sharding, and a simple RESTful API wrapped by clients in a growing number of languages. In this session, we will discuss the experience of developing and maintaining a RavenDB-backed CMS for one of the largest news sites in the US. We'll cover: - Supporting rapid evolution of the content/data model. - Indexing for full-text, map-reduce, geospatial and other types of search. - Replicating and sharding across servers and data centers for high-availability. - Deploying with no downtime. - Handling huge traffic spikes.

This document discusses web servers, including what they are, common features, differences between kernel-mode and user-mode servers, popular server software like Apache, IIS, Nginx, Google Web Server, and Resin. It also covers topics like path translation, load limits, overloads, and the market shares of different server products.

What is coming for VMware vSphere? Delivered at VMUG DK/UK/BE in November 2014. Session is all about vSphere futures, what can be expected in the near future.

This document discusses containers and Docker. It begins by explaining that cloud infrastructures comprise virtual resources like compute and storage nodes that are administered through software. Docker is introduced as a standard way to package code and dependencies into portable containers that can run anywhere. Key benefits of Docker include increased efficiency, consistency, and security compared to traditional virtual machines. Some weaknesses are that Docker may not be suitable for all applications and large container management can be difficult. Interesting uses of Docker include malware analysis sandboxes, isolating Skype sessions, and managing Raspberry Pi clusters with Docker Swarm.

HAProxy is a free, open-source load balancer and reverse proxy that is fast, reliable and offers high availability. It can be used to load balance HTTP and TCP-based applications. Some key features include out-of-band health checks, hot reconfiguration, and multiple load balancing algorithms. Many large companies use HAProxy to load balance their websites and applications. It runs on Linux, BSD, and Solaris and can be used to load balance applications across servers on-premises or in the cloud.

HAProxy is a free, open source load balancer and reverse proxy that is fast, reliable and offers high availability. It can be used to load balance HTTP and TCP traffic, provide failover, and maximize throughput and availability. Many large companies use HAProxy to load balance their websites and applications. It offers out of band health checks, hot reconfiguration, and works on Linux, BSD, Solaris and AIX. When implementing HAProxy, you configure backends, frontends and load balancing algorithms in its configuration file and can monitor it with tools like Nagios.

Software is eating infrastructure. Migrating reliability and scalability responsibilities up the stack from specialized hardware to software, cloud orchestration platforms such as Apache CloudStack (ACS) and object stores such as Riak CS increase the utilization and density of compute and storage resources by dynamically shifting workloads based on demand. Together, these platform can saturate compute and storage of 1000s of commodity hosts with strong operational visibility and end-user self-service. This presentation explores cloud design strategies to achieve high availability and reliability using commodity components. It then applies these strategies using Apache CloudStack and Riak CS.

This document discusses various techniques for improving WordPress performance through caching and deferred execution of code. It covers PHP caching using opcode caches, WordPress page caching plugins, WordPress object caching, and using Memcached. It also discusses deferred execution of code using asynchronous job queues to move non-critical tasks like email sending and push notifications out of the main request process.

The document discusses various strategies for achieving high availability of web applications and databases. It covers evaluating business requirements, DNS configuration, using cloud infrastructure or owning hardware, basic setups with application and database servers, database replication and clustering options, load balancing tools for Linux and cloud environments, auto scaling features, and monitoring. The key strategies presented include replicating databases, load balancing web traffic, auto-scaling cloud resources, and configuring failover between redundant application and database servers.

The document provides an agenda and overview of a session on hacking Apache CloudStack. The agenda includes introductions, a session on introducing CloudStack, and a hands-on session with DevCloud. The overview discusses what CloudStack is, how it works as an orchestration platform for IAAS clouds, its architecture and core components, and how users can consume and manage resources through it.

Where to start? - the first 2 hours of performance troubleshooting • The performance cheat sheet: cover all the basics before you start • Data collections and mining the logs • Common techniques to improve performance

The document provides an introduction to Docker, containers, and the problems they aim to solve. It discusses: - Why Docker was created - to address the "matrix from hell" of developing and deploying applications across different environments and platforms. - How Docker works at a high level, using lightweight containers that package code and dependencies to run consistently on any infrastructure. - Some key Docker concepts like images, containers, the Dockerfile for building images, and common Docker commands. - Benefits of Docker for developers and operations in simplifying deployment, reducing inconsistencies, and improving portability of applications.

Andy Gross from Basho discussed Riak Core, an open source distributed systems framework extracted from Riak. Riak Core provides abstractions like virtual nodes, preference lists, and event watchers to help developers build distributed applications. It is currently Erlang-only but will support other languages. Riak Core aims to allow developers to outsource complex distributed systems tasks and implement their own distributed systems more easily.

Not just for offline, JavaScript Service Workers give your web app a snappy response and predictable behavior. Your web app “feels like an app” to your more-satisfied users and stakeholders.

This document discusses real-time web applications and technologies. It defines real-time apps as allowing bi-directional communication between clients and servers so that users receive information as soon as it is published. Examples include chat, social media, gaming and notifications. Key implementation methods discussed are HTTP polling, streaming and WebSockets. The document also surveys popular real-time libraries for publish/subscribe, data syncing and hybrid approaches.

This document provides an overview of Locaweb's Infrastructure as a Service (IaaS) engineering team and capabilities. It discusses the team size and responsibilities, defines IaaS, describes Locaweb's IaaS architecture including virtual/physical servers, open source software used, provisioning processes, and resource monitoring. It also lists several open source projects developed by Locaweb to support their IaaS operations.

Presented at Gartner Data & Analytics, London Maty 2024. BT Group has used the Neo4j Graph Database to enable impressive digital transformation programs over the last 6 years. By re-imagining their operational support systems to adopt self-serve and data lead principles they have substantially reduced the number of applications and complexity of their operations. The result has been a substantial reduction in risk and costs while improving time to value, innovation, and process automation. Join this session to hear their story, the lessons they learned along the way and how their future innovation plans include the exploration of uses of EKG + Generative AI.

How do we build an IoT product, and make it profitable? Talk from the IoT meetup in March 2024. https://www.meetup.com/iot-sweden/events/299487375/

The presentation showcases the diverse real-world applications of Fused Deposition Modeling (FDM) across multiple industries: 1. **Manufacturing**: FDM is utilized in manufacturing for rapid prototyping, creating custom tools and fixtures, and producing functional end-use parts. Companies leverage its cost-effectiveness and flexibility to streamline production processes. 2. **Medical**: In the medical field, FDM is used to create patient-specific anatomical models, surgical guides, and prosthetics. Its ability to produce precise and biocompatible parts supports advancements in personalized healthcare solutions. 3. **Education**: FDM plays a crucial role in education by enabling students to learn about design and engineering through hands-on 3D printing projects. It promotes innovation and practical skill development in STEM disciplines. 4. **Science**: Researchers use FDM to prototype equipment for scientific experiments, build custom laboratory tools, and create models for visualization and testing purposes. It facilitates rapid iteration and customization in scientific endeavors. 5. **Automotive**: Automotive manufacturers employ FDM for prototyping vehicle components, tooling for assembly lines, and customized parts. It speeds up the design validation process and enhances efficiency in automotive engineering. 6. **Consumer Electronics**: FDM is utilized in consumer electronics for designing and prototyping product enclosures, casings, and internal components. It enables rapid iteration and customization to meet evolving consumer demands. 7. **Robotics**: Robotics engineers leverage FDM to prototype robot parts, create lightweight and durable components, and customize robot designs for specific applications. It supports innovation and optimization in robotic systems. 8. **Aerospace**: In aerospace, FDM is used to manufacture lightweight parts, complex geometries, and prototypes of aircraft components. It contributes to cost reduction, faster production cycles, and weight savings in aerospace engineering. 9. **Architecture**: Architects utilize FDM for creating detailed architectural models, prototypes of building components, and intricate designs. It aids in visualizing concepts, testing structural integrity, and communicating design ideas effectively. Each industry example demonstrates how FDM enhances innovation, accelerates product development, and addresses specific challenges through advanced manufacturing capabilities.

This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator. Link to presentation recording and slides: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/ Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.

Have you noticed the OpenSSF Scorecard badges on the official Dart and Flutter repos? It's Google's way of showing that they care about security. Practices such as pinning dependencies, branch protection, required reviews, continuous integration tests etc. are measured to provide a score and accompanying badge. You can do the same for your projects, and this presentation will show you how, with an emphasis on the unique challenges that come up when working with Dart and Flutter. The session will provide a walkthrough of the steps involved in securing a first repository, and then what it takes to repeat that process across an organization with multiple repos. It will also look at the ongoing maintenance involved once scorecards have been implemented, and how aspects of that maintenance can be better automated to minimize toil.

Invited Remote Lecture to SC21 The International Conference for High Performance Computing, Networking, Storage, and Analysis St. Louis, Missouri November 18, 2021

The integration of programming into civil engineering is transforming the industry. We can design complex infrastructure projects and analyse large datasets. Imagine revolutionizing the way we build our cities and infrastructure, all by the power of coding. Programming skills are no longer just a bonus—they’re a game changer in this era. Technology is revolutionizing civil engineering by integrating advanced tools and techniques. Programming allows for the automation of repetitive tasks, enhancing the accuracy of designs, simulations, and analyses. With the advent of artificial intelligence and machine learning, engineers can now predict structural behaviors under various conditions, optimize material usage, and improve project planning.

Kief Morris rethinks the infrastructure code delivery lifecycle, advocating for a shift towards composable infrastructure systems. We should shift to designing around deployable components rather than code modules, use more useful levels of abstraction, and drive design and deployment from applications rather than bottom-up, monolithic architecture and delivery.

To help you choose the best DiskWarrior alternative, we've compiled a comparison table summarizing the features, pros, cons, and pricing of six alternatives.

An invited talk given by Mark Billinghurst on Research Directions for Cross Reality Interfaces. This was given on July 2nd 2024 as part of the 2024 Summer School on Cross Reality in Hagenberg, Austria (July 1st - 7th)

YOUR RELIABLE WEB DESIGN & DEVELOPMENT TEAM — FOR LASTING SUCCESS WPRiders is a web development company specialized in WordPress and WooCommerce websites and plugins for customers around the world. The company is headquartered in Bucharest, Romania, but our team members are located all over the world. Our customers are primarily from the US and Western Europe, but we have clients from Australia, Canada and other areas as well. Some facts about WPRiders and why we are one of the best firms around: More than 700 five-star reviews! You can check them here. 1500 WordPress projects delivered. We respond 80% faster than other firms! Data provided by Freshdesk. We’ve been in business since 2015. We are located in 7 countries and have 22 team members. With so many projects delivered, our team knows what works and what doesn’t when it comes to WordPress and WooCommerce. Our team members are: - highly experienced developers (employees & contractors with 5 -10+ years of experience), - great designers with an eye for UX/UI with 10+ years of experience - project managers with development background who speak both tech and non-tech - QA specialists - Conversion Rate Optimisation - CRO experts They are all working together to provide you with the best possible service. We are passionate about WordPress, and we love creating custom solutions that help our clients achieve their goals. At WPRiders, we are committed to building long-term relationships with our clients. We believe in accountability, in doing the right thing, as well as in transparency and open communication. You can read more about WPRiders on the About us page.

Manual Method of Product Research | Helium10 | MBS RETRIEVER

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr "A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models" arXiv2023 https://arxiv.org/abs/2307.12980

Everything that I found interesting about engineering leadership last month

Is your patent a vanity piece of paper for your office wall? Or is it a reliable, defendable, assertable, property right? The difference is often quality. Is your patent simply a transactional cost and a large pile of legal bills for your startup? Or is it a leverageable asset worthy of attracting precious investment dollars, worth its cost in multiples of valuation? The difference is often quality. Is your patent application only good enough to get through the examination process? Or has it been crafted to stand the tests of time and varied audiences if you later need to assert that document against an infringer, find yourself litigating with it in an Article 3 Court at the hands of a judge and jury, God forbid, end up having to defend its validity at the PTAB, or even needing to use it to block pirated imports at the International Trade Commission? The difference is often quality. Quality will be our focus for a good chunk of the remainder of this season. What goes into a quality patent, and where possible, how do you get it without breaking the bank? ** Episode Overview ** In this first episode of our quality series, Kristen Hansen and the panel discuss: ⦿ What do we mean when we say patent quality? ⦿ Why is patent quality important? ⦿ How to balance quality and budget ⦿ The importance of searching, continuations, and draftsperson domain expertise ⦿ Very practical tips, tricks, examples, and Kristen’s Musts for drafting quality applications https://www.aurorapatents.com/patently-strategic-podcast.html

Cybersecurity is a major concern in today's connected digital world. Threats to organizations are constantly evolving and have the potential to compromise sensitive information, disrupt operations, and lead to significant financial losses. Traditional cybersecurity techniques often fall short against modern attackers. Therefore, advanced techniques for cyber security analysis and anomaly detection are essential for protecting digital assets. This blog explores these cutting-edge methods, providing a comprehensive overview of their application and importance.

Sustainability requires ingenuity and stewardship. Did you know Pigging Solutions pigging systems help you achieve your sustainable manufacturing goals AND provide rapid return on investment. How? Our systems recover over 99% of product in transfer piping. Recovering trapped product from transfer lines that would otherwise become flush-waste, means you can increase batch yields and eliminate flush waste. From raw materials to finished product, if you can pump it, we can pig it.

These fighter aircraft have uses outside of traditional combat situations. They are essential in defending India's territorial integrity, averting dangers, and delivering aid to those in need during natural calamities. Additionally, the IAF improves its interoperability and fortifies international military alliances by working together and conducting joint exercises with other air forces.

The DealBook is our annual overview of the Ukrainian tech investment industry. This edition comprehensively covers the full year 2023 and the first deals of 2024.

Your comprehensive guide to RPA in healthcare for 2024. Explore the benefits, use cases, and emerging trends of robotic process automation. Understand the challenges and prepare for the future of healthcare automation