Krux operates a large infrastructure serving thousands of user requests per second. They use Puppet and tools like Cloudkick, Foreman, Boto, and Vagrant to manage their infrastructure in an automated and scalable way. Their Puppet configuration is split into modules, environments, and datacenters. They launch AWS nodes programmatically and configure them with Puppet. Cloudkick is used for monitoring and parallel SSH. Boto allows full Python API access to AWS. Vagrant allows consistently provisioning development machines locally. Automation and external configuration enable their small operations team to manage a large, dynamic infrastructure.
Drupal VM for Drupal 8 Dev - Drupal Camp STL 2017Jeff Geerling
Drupal VM is a VM for Drupal development, built with Vagrant and Ansible, or Docker. This presentation was given at Drupal Camp St. Louis 2017 by Jeff Geerling, Senior Technical Architect at Acquia.
Practical Operation Automation with StackStormShu Sugimoto
Automation is getting more and more important these days, but it is not always easy to achieve, because it requires tremendous effort to convert existing procedures machine-friendly. That often means, you need to change almost everything!
StackStorm (aka st2, https://stackstorm.com/) is an open source IFTTT-ish middleware that ships with powerful workflow engine and unique features called "inquiries".
I'll focus on this workflow engine functionalities of st2 and show how these can ease the "automation" of day to day tasks. The example I'll show in this presentation is the actual workflow that we use at JPNAP, the real world IXP operation.
Configuration management and deployment with ansibleIvan Dimitrov
This document discusses configuration management and deployment using Ansible. It provides an overview of Ansible's key capabilities including configuration management, ad-hoc execution, rolling updates, multi-tier deployment and cloud provisioning. It also discusses Ansible's architecture including being SSH-based and agentless, its use of inventory files, playbooks and roles to define configurations. Finally it demonstrates examples of using Ansible modules, tasks, and playbooks to manage and deploy applications.
My talk Qual'�� la prima cosa che ti viene in mente se ti chiedessi di associare "container" e "architetture" ? Esatto ! Container e microservizi sono di fatto gli argomenti più chiacchierati degli ultimi anni e che spesso vanno a braccetto. Ma quand'è che i due mondi iniziano ad incontrarsi, se non nel tuo computer? Ti porterò la mia esperienza sul uso dei container per lo sviluppo di un applicativo con architettura a microservizi, partendo dalle scelte fatte fin dai primi momenti dello sviluppo fino ad arrivare alla sua messa in produzione.
The document discusses the importance of resilience and failure preparation in cloud systems. It introduces the concept of "GameDay", which are large-scale exercises that inject faults across critical systems to increase an organization's resilience. These exercises help identify risks, reduce failure frequency and recovery time, and build confidence and competence in responding to failures. The document advocates designing infrastructure to be rebuildable from source code and data backups alone. It also stresses the importance of automation, emergencies tools testing, and culture of collaboration between dev and ops.
This document summarizes updates and changes between Spring Boot 1 and Spring Boot 2, including updates to Gradle, Spring Cloud Config Server, Spring Web, Spring Data, Spring Data JPA, Flyway, testing, and configuration property binding. It provides details on specific changes such as dependency and API changes.
DevOps for Humans - Ansible for Drupal Deployment Victory!Jeff Geerling
Everyone knows it's a Good Idea™ to use a configuration management system (e.g. Puppet, Chef) to manage your Drupal infrastructure. But many people (myself included) have run into a wall of #wtfmoments when trying to learn the vagaries of traditional CM systems and their vendor-specific syntaxes.
In 2012, Ansible was released, enabling normal human beings to manage their servers with an easy, but powerful, CM system that uses YAML (just like Drupal 8!) to define configuration and Jinja2 (very much like Twig!) for templates. Not only that, but Ansible is also an incredibly simple and very flexible Drupal deployment and continuous delivery tool.
Learn how you can use Ansible to manage your infrastructure—including local development environments—and stop letting servers and deployments get in the way of development.
Drupal VM for Drupal 8 Dev - MidCamp 2017Jeff Geerling
These slides were used in my presentation "Developing for Drupal 8 with Drupal VM", given at MidCamp in Chicago, IL on 2017-04-01.
Learn how to build a modern Drupal 8 website using Composer and Drupal VM for local and prod!
This document configures a Spring Boot application to export metrics to Amazon CloudWatch using Micrometer. It sets the CloudWatch namespace, disables auto-configuration of AWS stack and region, and statically defines the region as ap-northeast-1. It also defines a HeapMemoryUsage gauge that will report JVM heap memory usage to the metrics registry.
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLAsean_seannery
This is a talk that was given for the Scalable Internet Services Masters-level Computer Science class at UCLA and UCSB. It briefly discusses the server architecture for the game League of Legends before going into depth about how the data warehouse can hold petabytes of player data. Discussion about message queue architecture and scalability occurs along the way
React is a UI library that is changing the way web applications are written. While there are many benefits to using React, managing an application's complexity as it scales is one of the most powerful.
Achieving Continuous Delivery: An Automation Storyjimi-c
Continuos Deployment is the act of deploying software constantly. The idea is if "release early, release often" is good, releasing very often is better. It's not trivial. Automation is part of the battle, and testing is another. Learn to use tools like Jenkins and Ansible to move from deploying software once a month to 15 times every hour, and why you'll want to.
Presented at PyCon 2015 in Montreal
Chasing AMI - Building Amazon machine images with Puppet, Packer and JenkinsTomas Doran
Using puppet when configuring EC2 machines seems a natural fit. However bringing up new machines from a community image with puppet is not trivial and can be slow, and so not useful for auto-scaling.
The cloud also offers a solution to ongoing server maintenance, allowing you to launch fresh instances whenever you upgrade your applications (Immutable or Phoenix servers). However to predictably succeed, you need to freeze the puppet code alongside the application version for deployment.
The solution to these issues is generating custom machine images (AMIs) with your software inlined. This talk will cover Yelp's use of a Packer, Jenkins and Puppet for generating AMIs. This will include how we deal with issues like bootstrapping, getting canonical information about a machine's environment and cluster state at launch time, as well as supporting immutable/phoenix servers in combination with more traditional long lived servers inside our hybrid cloud infrastructure.
Windows Azure Web Sites - Things they don’t teach kids in school - BuildStuffLTMaarten Balliauw
Microsoft has a cloud platform which runs .NET, NodeJS and PHP. All 101 talks out there will show you the same: it’s easy to deploy, it scales out on demand and it runs WordPress. Great! But what about doing real things? In this session, we’ll explore the things they don’t teach kids in school. How about trying to find out the architecture of this platform? What about the different deployment options for Windows Azure Web Sites, the development flow and some awesome things you can do with the command line tools? Did you know you can modify the automated build process? Join me in this exploration of some lesser known techniques of the platform.
ShapeBlue is an expert company in building public and private clouds. They specialize in deploying the open source CloudStack cloud infrastructure software and surrounding systems. The document discusses Network Functions Virtualization (NFV), how it relates to CloudStack's existing virtual router capabilities, and some potential future enhancements to CloudStack to better support NFV use cases and topologies. These include adding new network types like layer 2 networks, improving support for multi-device topologies through a new "Enterprise Topology" virtual router, and integrating virtualized network functions as configurable network providers.
Blue/Green deployments have been an important, if rarely implemented, technique in the Continuous Delivery playbook for years. Their aim is simple: provision, deploy, test — and optionally roll-back — your application before it's served to the public. Betterment's deployment architecture takes a similar, but more straightforward approach, accomplishing the important goals sought out by Blue/Green practitioners. Dubbed 'Cyan' (a mixture of Blue/Green), Betterment uses Ansible to provision new instances, push the latest artifacts to them, and ensure that they're healthy before marking them ready for production. All this ensures fast, stable, zero-downtime rollout with minimal human interaction. We'll discuss Betterment's philosophical approach to shipping new code and then dive into the nitty-gritty Ansible that powers the whole thing.
Breaking Up With Your Data Center PresentationTelescope_Inc
Telescope Inc.'s presentation at AnsibleFest conference. In this talk, discuss how they moved from a baremetal datacenter to AWS, things learned along the way, and how they scale up to meet voting demand with Ansible.
Bare Metal to OpenStack with Razor and ChefMatt Ray
Razor is an open source provisioning tool that was originally developed by EMC and Puppet Labs. It can discover hardware, select images to deploy, and provision nodes using model-based provisioning. The demo showed setting up a Razor appliance, adding images, models, policies, and brokers. It then deployed an OpenStack all-in-one environment to a new VM using Razor and Chef. The OpenStack cookbook walkthrough explained the roles, environments, and cookbooks used to deploy and configure OpenStack components using Chef.
- Puppet is a tool that allows you to define infrastructure as code and manage your infrastructure in a consistent, versioned way like software. It uses resources and relationships between them to define what state should exist.
- Modules contain Puppet code, files, and templates to manage specific components like Apache or MySQL. They can be found on the Puppet Forge or GitHub.
- Puppet uses a dependency graph of resources to determine the order of operations needed to ensure resources are in the desired state. It is highly customizable through Ruby plugins.
- The autoami module contains tools to automatically generate new AMIs when Puppet runs detect configuration changes, ensuring infrastructure is always up to date. It uses custom Puppet faces
This document provides instructions for building an HPC cluster on AWS using the cfnCluster tool in 10 minutes. It discusses establishing an AWS account, pulling the cfnCluster source code from GitHub, generating SSH keys, configuring cfnCluster, and other configuration options to consider like using low-cost t2.micro instance types when first experimenting with cfnCluster functionality. The overall process demonstrated allows provisioning an HPC cluster within AWS that includes a head node and auto-scaling compute nodes connected over a 10G network, using CloudFormation templates managed by cfnCluster.
This talk will focus on a brief overview of Kubernetes, with a brief demo, and then more of an in-depth focus on issues we've faced moving PHP projects into Docker and Kubernetes like signal propagation, init systems, and logging.
Talk from Cape Town PHP meetup on Feb. 7, 2016:
https://www.meetup.com/Cape-Town-PHP-Group/events/237226310/
Code: https://github.com/zoidbergwill/kubernetes-php-examples
Slides as markdown: http://www.zoidbergwill.com/presentations/2017/kubernetes-php/index.md
Get hands-on with security features and best practices to protect your containerized services. Learn to push and verify signed images with Docker Content Trust, and collaborate with delegation roles. Intermediate to advanced level Docker experience recommended, participants will be building and pushing with Docker during the workshop.
Led By Docker Security Experts:
Riyaz Faizullabhoy
David Lawrence
Viktor Stanchev
Experience Level: Intermediate to advanced level Docker experience recommended
A presentation I gave on September 26 at the Melbourne Symfony developers group on using Environment Variables (envvars) in Symfony and managing secrets in your PHP applications.
For more information on these subjects, check out the supporting piece I wrote: https://samjarrett.com.au/swipe-right
This document provides instructions for setting up a small 3 node Hadoop cluster on Amazon EC2. It covers configuring EC2 instances, installing Java and Hadoop, configuring the Hadoop nodes and services, and running a sample MapReduce job to validate the cluster. The goal is to provide a simple tutorial for getting started with Hadoop on EC2 for learning purposes.
Writing & Sharing Great Modules - Puppet Camp BostonPuppet
This document provides best practices and guidance for writing and sharing Puppet modules. It discusses separating logic from data, using semantic versioning (SemVer), creating modules as interfaces, reusing existing modules from the Puppet Forge, and establishing a community to collaborate on modules. The key recommendations are to separate configuration data from logic, use SemVer to avoid breaking changes, make modules opinionated but allow overrides, leverage existing modules, and engage the community to improve modules.
Listen up, developers. You are not special. Your infrastructure is not a beautiful and unique snowflake. You have the same tech debt as everyone else. This is a talk about a better way to build and manage infrastructure: Terraform Modules. It goes over how to build infrastructure as code, package that code into reusable modules, design clean and flexible APIs for those modules, write automated tests for the modules, and combine multiple modules into an end-to-end techs tack in minutes.
You can find the video here: https://www.youtube.com/watch?v=LVgP63BkhKQ
Tuesday, July 30th session of the vBrownBag OpenStack Sack Lunch Series: Couch to OpenStack. We cover Nova, the Compute Service that deploys and runs VMs.
This document discusses using Puppet and infrastructure as code to manage Apache CloudStack infrastructure. It introduces the cloudstack_resources Puppet module which allows defining CloudStack instances and entire application stacks in Puppet manifests. This enables treating infrastructure like code where Puppet can deploy and configure entire environments on CloudStack. Examples are given of classifying servers and deploying a Hadoop cluster with a single Puppet resource definition. Links are provided to resources for using Puppet with CloudStack and videos that further explain the concepts.
This document provides an overview of Kubernetes and attacking Kubernetes clusters for penetration testers. It begins with introductions to containers, Kubernetes, and setting up a local Kubernetes cluster. It then covers a threat model for Kubernetes and describes an attacker's workflow against a cluster, including discovery, vulnerability testing, exploitation, and persistence. Specific attacks demonstrated include API server authorization testing, discovering exposed etcd and internal services, container escapes, and Helm Tiller privilege escalation. Resources for further learning are also provided.
This document provides step-by-step instructions for building a simple orchestrator. It begins by setting up the basic components including RabbitMQ as a messaging broker and Celery workers. Step 1 demonstrates executing a simple AWS resource by adding a task to the queue. Step 2 adds a MongoDB database to store resources outside of the queue. Step 3 builds a service level on top by allowing resources like AWS instances and Docker containers to be orchestrated together through a YAML file.
Presentation at March 2019 Dutch Postgres User Group Meetup on lessons learnt while migrating from Oracle to Postgres, demo'ed via vagrant test environments and using generic pgbench datasets.
The document discusses running memcached clusters on Amazon EC2. It covers key concepts like caching, clusters, and infrastructure as a service (AWS). It then provides step-by-step instructions for setting up a memcached cluster on EC2, including creating security groups, launching EC2 instances from AMIs, and configuring the memcached servers and clients. The summary concludes that setting up and running memcached clusters on infrastructure as a service environments like EC2 is straightforward.
This document discusses using Puppet to manage infrastructure as code with Apache CloudStack. It describes how Puppet types and providers were developed to allow defining CloudStack instances and entire application stacks in Puppet manifests. This enables automated deployment and configuration of infrastructure along with software configuration. Examples are given of using Puppet to define CloudStack instances, groups of instances that make up an application stack, and setting defaults for attributes. Resources mentioned include the CloudStack and Puppet GitHub pages.
June 2014 PDX PUG: Writing and Publishing Puppet Modules Puppet
The document discusses how to write and publish Puppet modules. It explains that a Puppet module encapsulates configuration for a service and includes code, files, templates, tests, and documentation. It provides guidance on module structure and content, strategies for parameterizing templates, testing modules, and publishing modules to the Forge for others to use.
Build Your Own CaaS (Container as a Service)HungWei Chiu
In this slide, I introduce the kubernetes and show an example what is CaaS and what it can provides.
Besides, I also introduce how to setup a continuous integration and continuous deployment for the CaaS platform.
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionBert Blevins
Cybersecurity is a major concern in today's connected digital world. Threats to organizations are constantly evolving and have the potential to compromise sensitive information, disrupt operations, and lead to significant financial losses. Traditional cybersecurity techniques often fall short against modern attackers. Therefore, advanced techniques for cyber security analysis and anomaly detection are essential for protecting digital assets. This blog explores these cutting-edge methods, providing a comprehensive overview of their application and importance.
Sustainability requires ingenuity and stewardship. Did you know Pigging Solutions pigging systems help you achieve your sustainable manufacturing goals AND provide rapid return on investment.
How? Our systems recover over 99% of product in transfer piping. Recovering trapped product from transfer lines that would otherwise become flush-waste, means you can increase batch yields and eliminate flush waste. From raw materials to finished product, if you can pump it, we can pig it.
Kief Morris rethinks the infrastructure code delivery lifecycle, advocating for a shift towards composable infrastructure systems. We should shift to designing around deployable components rather than code modules, use more useful levels of abstraction, and drive design and deployment from applications rather than bottom-up, monolithic architecture and delivery.
The Rise of Supernetwork Data Intensive ComputingLarry Smarr
Invited Remote Lecture to SC21
The International Conference for High Performance Computing, Networking, Storage, and Analysis
St. Louis, Missouri
November 18, 2021
Transcript: Details of description part II: Describing images in practice - T...BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and slides: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
The DealBook is our annual overview of the Ukrainian tech investment industry. This edition comprehensively covers the full year 2023 and the first deals of 2024.
Quality Patents: Patents That Stand the Test of TimeAurora Consulting
Is your patent a vanity piece of paper for your office wall? Or is it a reliable, defendable, assertable, property right? The difference is often quality.
Is your patent simply a transactional cost and a large pile of legal bills for your startup? Or is it a leverageable asset worthy of attracting precious investment dollars, worth its cost in multiples of valuation? The difference is often quality.
Is your patent application only good enough to get through the examination process? Or has it been crafted to stand the tests of time and varied audiences if you later need to assert that document against an infringer, find yourself litigating with it in an Article 3 Court at the hands of a judge and jury, God forbid, end up having to defend its validity at the PTAB, or even needing to use it to block pirated imports at the International Trade Commission? The difference is often quality.
Quality will be our focus for a good chunk of the remainder of this season. What goes into a quality patent, and where possible, how do you get it without breaking the bank?
** Episode Overview **
In this first episode of our quality series, Kristen Hansen and the panel discuss:
⦿ What do we mean when we say patent quality?
⦿ Why is patent quality important?
⦿ How to balance quality and budget
⦿ The importance of searching, continuations, and draftsperson domain expertise
⦿ Very practical tips, tricks, examples, and Kristen’s Musts for drafting quality applications
https://www.aurorapatents.com/patently-strategic-podcast.html
INDIAN AIR FORCE FIGHTER PLANES LIST.pdfjackson110191
These fighter aircraft have uses outside of traditional combat situations. They are essential in defending India's territorial integrity, averting dangers, and delivering aid to those in need during natural calamities. Additionally, the IAF improves its interoperability and fortifies international military alliances by working together and conducting joint exercises with other air forces.
論文紹介:A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...Toru Tamaki
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr "A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models" arXiv2023
https://arxiv.org/abs/2307.12980
Details of description part II: Describing images in practice - Tech Forum 2024BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and transcript: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
UiPath Community Day Kraków: Devs4Devs ConferenceUiPathCommunity
We are honored to launch and host this event for our UiPath Polish Community, with the help of our partners - Proservartner!
We certainly hope we have managed to spike your interest in the subjects to be presented and the incredible networking opportunities at hand, too!
Check out our proposed agenda below 👇👇
08:30 ☕ Welcome coffee (30')
09:00 Opening note/ Intro to UiPath Community (10')
Cristina Vidu, Global Manager, Marketing Community @UiPath
Dawid Kot, Digital Transformation Lead @Proservartner
09:10 Cloud migration - Proservartner & DOVISTA case study (30')
Marcin Drozdowski, Automation CoE Manager @DOVISTA
Pawel Kamiński, RPA developer @DOVISTA
Mikolaj Zielinski, UiPath MVP, Senior Solutions Engineer @Proservartner
09:40 From bottlenecks to breakthroughs: Citizen Development in action (25')
Pawel Poplawski, Director, Improvement and Automation @McCormick & Company
Michał Cieślak, Senior Manager, Automation Programs @McCormick & Company
10:05 Next-level bots: API integration in UiPath Studio (30')
Mikolaj Zielinski, UiPath MVP, Senior Solutions Engineer @Proservartner
10:35 ☕ Coffee Break (15')
10:50 Document Understanding with my RPA Companion (45')
Ewa Gruszka, Enterprise Sales Specialist, AI & ML @UiPath
11:35 Power up your Robots: GenAI and GPT in REFramework (45')
Krzysztof Karaszewski, Global RPA Product Manager
12:20 🍕 Lunch Break (1hr)
13:20 From Concept to Quality: UiPath Test Suite for AI-powered Knowledge Bots (30')
Kamil Miśko, UiPath MVP, Senior RPA Developer @Zurich Insurance
13:50 Communications Mining - focus on AI capabilities (30')
Thomasz Wierzbicki, Business Analyst @Office Samurai
14:20 Polish MVP panel: Insights on MVP award achievements and career profiling
Coordinate Systems in FME 101 - Webinar SlidesSafe Software
If you’ve ever had to analyze a map or GPS data, chances are you’ve encountered and even worked with coordinate systems. As historical data continually updates through GPS, understanding coordinate systems is increasingly crucial. However, not everyone knows why they exist or how to effectively use them for data-driven insights.
During this webinar, you’ll learn exactly what coordinate systems are and how you can use FME to maintain and transform your data’s coordinate systems in an easy-to-digest way, accurately representing the geographical space that it exists within. During this webinar, you will have the chance to:
- Enhance Your Understanding: Gain a clear overview of what coordinate systems are and their value
- Learn Practical Applications: Why we need datams and projections, plus units between coordinate systems
- Maximize with FME: Understand how FME handles coordinate systems, including a brief summary of the 3 main reprojectors
- Custom Coordinate Systems: Learn how to work with FME and coordinate systems beyond what is natively supported
- Look Ahead: Gain insights into where FME is headed with coordinate systems in the future
Don’t miss the opportunity to improve the value you receive from your coordinate system data, ultimately allowing you to streamline your data analysis and maximize your time. See you there!
Measuring the Impact of Network Latency at TwitterScyllaDB
Widya Salim and Victor Ma will outline the causal impact analysis, framework, and key learnings used to quantify the impact of reducing Twitter's network latency.
Best Programming Language for Civil EngineersAwais Yaseen
The integration of programming into civil engineering is transforming the industry. We can design complex infrastructure projects and analyse large datasets. Imagine revolutionizing the way we build our cities and infrastructure, all by the power of coding. Programming skills are no longer just a bonus—they’re a game changer in this era.
Technology is revolutionizing civil engineering by integrating advanced tools and techniques. Programming allows for the automation of repetitive tasks, enhancing the accuracy of designs, simulations, and analyses. With the advent of artificial intelligence and machine learning, engineers can now predict structural behaviors under various conditions, optimize material usage, and improve project planning.
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfNeo4j
Presented at Gartner Data & Analytics, London Maty 2024. BT Group has used the Neo4j Graph Database to enable impressive digital transformation programs over the last 6 years. By re-imagining their operational support systems to adopt self-serve and data lead principles they have substantially reduced the number of applications and complexity of their operations. The result has been a substantial reduction in risk and costs while improving time to value, innovation, and process automation. Join this session to hear their story, the lessons they learned along the way and how their future innovation plans include the exploration of uses of EKG + Generative AI.
11. Our Traffic
• Serving 4000-10000 user & contextual data
requests/second
• Sub 100 ms response times
• Processing ~150 gb of raw data per day
• Twitter: Average ~3000 tweets/second
12. Our Infrastructure
• Started small on AWS. Now:
• 100 dedicated nodes
• +100-200 on demand Map/Reduce nodes
• Dozens of local development machines
• 20 different types of machines
19. cloud-init
Uses AMI user-data to bootstrap puppet on the client
https://help.ubuntu.com/community/CloudInit
http://www.youtube.com/watch?v=-zL3BdbKyGY
22. you can upgrade
the kernel
Only AMI that I know that can do this
http://cloud.ubuntu.com/2011/02/migrating-to-pv-
grub-kernels-for-kernel-upgrades/
23. Updated software for
10.04
Backported builds for
Apache, Memcache, Mysql, PHP, etc
https://launchpad.net/~ubuntu-server-edgers
26. <3 Elastic Load
Balancer
They're free and will save you more than once
http://aws.amazon.com/elasticloadbalancing/
27. <3 S3
(Simple Storage Service)
Great cheap data retention
Good poor mans CDN
http://aws.amazon.com/s3
28. Tip: Get ExpanDrive for
great SSHFS and S3FS
Available for Windows and Mac:
http://www.expandrive.com/
29. RDS > Own MySQL
Hot Standby - Failover is ~7 minutes
Read Replicates - Improve read performance
BUT, you can't replicate out of RDS :(
http://aws.amazon.com/rds/
30. Use EBS Root
(Elastic Block Storage)
You can reboot and stop/start machines and keep state
Consider attaching extra EBS for data persistence
Tip: Software raid for multiple EBS drives for better IO
31. </3 Network
Partitioning
This will happen to you a lot
Relying on network connections will decrease
availability of your machines
32. </3 Floating
public IPS
AWS DHCP server is flaky
AWS DNS TTL is 60 seconds
Limited amount of fixed public IPs
33. Sort your DNS
AWS offers http://aws.amazon.com/route53/
When you go multi data center or have big traffic,
seriously consider Dyn: http://dyn.com/dns/
34. Avoid Single
Points of Failure
Because they WILL fail.
Architect for eventually consistent,
distributed systems where you can.
37. Optimize for making
Puppet development
EASY
Bridge the gap between dev & ops
Tip: use a c1.medium at least
38. Put your Puppet
code in VCS
I really don't need to explain why, right?
39. Run multiple Puppet
environments
http://docs.puppetlabs.com/guides/environment.html
We put 1 host of each cluster in puppet environment
development, 1 in staging, the rest in production
Don't break everything at once :)
40. Split your Puppet
code into modules
We use: Forge, Components, Services
http://docs.puppetlabs.com/guides/modules.html
41. Use seperate init.pp,
params.pp & config.pp
Params.pp so you can include variables from elsewhere
Config.pp lets you specify:
kfoo::config { $fqdn } in a service
and require:
Kfoo::Config[ $fqdn ] in the component
http://docs.puppetlabs.com/guides/modules.html
42. Use a common
base class
Set up all the plumbing from users, to apt,
to filesystems, to mounts, ntp, sudo, git,
monitoring, ssh, and so on.
Run it early using run stages
43. Sample Service
class s_webui {
include kbase
include kapache
include kwebui
include kredis
kwebui { $fqdn: }
kapache::vhost { $fqdn: ssl => 443 }
kredis::config { $fqdn: memory => '100M' }
}
44. Write tools to make
you more productive
Enable developers to run their own Puppet master
Create new components easily
Push changes to production
Our code: https://github.com/krux/ops-tools /
45. Your own Puppet server
& manifests
puppet001:puppet-jib$ screen -S jib.puppetmaster
bin/run_puppet_master_locally 8180
Running: sudo puppet master --no-daemonize
--verbose --debug --masterport 8180
--pidfile /mnt/tmp/puppetmaster.8180.pid
--confdir /data/git/puppet-jib/bin/..
.....
notice: Starting Puppet master version 2.6.3
.....
47. Use an External
Node Classifier
Manage your host specific configuration
separately from your manifests
http://docs.puppetlabs.com/guides/external_nodes.html
Our code: https://github.com/krux/ops-tools /blob/puppet/bin/node_classifier.py
48. Keep node
configuration in an
editable location
We chose S3
Git, LDAP, or anything else that works for you.
49. Sign nodes that have
a configuration only
Keyed off their certname, run periodically
Inspired by:
http://ubuntumathiaz.wordpress.com/2010/03/24/using-
puppet-in-uecec2-puppet-support-in-ubuntu-images/
Our code: https://github.com/krux/ops-tools /blob/puppet/bin/check_csr.py
55. Email
Reports & Alerts
This feature alone is worth installing it.
Run it on the same host as your
Puppet master for minimal friction
http://theforeman.org/projects/foreman/wiki/
Summarized_E-Mail_Reports
57. Theoretically:
Node Classifier
http://theforeman.org/projects/foreman/wiki/
External_Nodes
We are happy with S3 based solution
YMMV though: do look into it!
60. $ s3cmd put file.txt
s3://my-bucket
Great for cronjobs, maintenance tasks & file syncs
Consider s3://my-dropbox for your company
http://s3tools.org/s3cmd
61. boto: Full python API
access to AWS
Boto + AWS + Puppet
=
Real 'Infrastructure as Code'
http://code.google.com/p/boto/
62. start_instance.py:
Launch AWS nodes
Manage zone, security group, type ami,
puppet class, EBS, hostname
Bootstraps the node for puppet,
integrates with external node classifier
Our code: https://github.com/krux/ops-tools /blob/aws/bin/start_instance.py
63. $ start_instance.py -t m1.large -z us-east-1a -a 10
-H dev001.example.com -s mycorp-development
ami-2ec83147 s_development
Starting instance of ami ami-2ec83147 - this may take a while
......... started i-12345678
Attaching 10gb volume to instance i-12345678 - this may take a while
..... attached vol-87654321
Created these DNS entries:
dev001.example.com => ec2-172-131-213-58.compute-1.amazonaws.com
Wrote configuration to S3 key:
s3://instances/dev001.example.com.47334fd8-1516-451d-bd5a-8760ab2a36c0
64. security_groups.py
Manage & Sync
Programmatically manage your security groups
keep groups in sync across regions
Our code: https://github.com/krux/ops-tools /blob/aws/bin/security_groups.py
66. Free developer
account
1 Free node with all features,
unlimited nodes with basic features
Free: HTTP(S), PING, SSH, DNS, TCP
Premium: HTTP JSON(!), Custom plugins, Mysql, Apache
mod_status, etc.
Get a 2nd free node through referral:
https://cloudkick.com/referral/633f0729
69. Generate your
cloudkick.conf from
Puppet
Use puppet classes, tags, colors as you define them
as cloudkick tags
Our code for doing so: https://gist.github.com/1230044
70. Cloudkick Gem for
parallel-ssh
Uses your cloudkick tags to do node selection,
which are based straight off your puppet classes & facts
https://github.com/cloudkick/cloudkick-gem
72. Krux Improvements:
pscp, listing nodes
Get it from our github:
https://github.com/krux/cloudkick-gem
Fork and contribute!
73. Cloudkick list
$cloudkick list --full --query 'node:redis-c*'
# Name IP Type Zone
redis-c-master001 52.13.118.158 m2.4xlarge us-east-1a
redis-c-slave001 64.206.11.221 m2.4xlarge us-east-1a
redis-c-slave002 183.71.131.32 m2.4xlarge us-east-1b
redis-c-slave004 52.16.34.217 m2.4xlarge us-east-1d
74. Take away:
Measure Everything!
Further reading:
Pagerduty for cell phone/pager/email alerts
New Relic for more in depth app monitoring
MCollective for more advanced task parallelization
77. VirtualBox + Ubuntu
+ Puppet = JFDI
Use same puppet infrastructure to provision
dev machines locally
Put it on a USB stick, be up and running in 30 minutes
Our code for doing so: https://gist.github.com/1230221