How to Puppetize Google Cloud Platform - PuppetConf 2014

HOWTO:Puppetizing Google Cloud Platform
Katharina Probst, Technical Lead & Manager (Google)
Ryan Coleman, Product Manager (Puppet Labs)

Agenda: September 23rd, 2014
Whirlwind tour of Google Cloud Platform
Diving into Google Compute Engine
How Puppet Labs uses Google Cloud Platform
Live Demo
1
2
3
4

For the past 15 years, Google
has been building out the
world’s fastest, most powerful,
highest quality cloud
infrastructure on the planet.
Images by Connie
Zhou
Why Google Cloud Platform?

Building Products that Scale
Google Maps Gmail Google Drive YouTube

A Network that Spans the Globe

Innovating Software & Driving Technology Forward
MapReduce Dremel Spanner
Big Table Colossus
GFS
Compute
Engine
2002 2004 2006 2008 2010 2012 2013

“[Google's] ability to build, organize, and
operate a huge network of servers and fiber-optic
cables with an efficiency and speed that
rocks physics on its heels.
This is what makes Google Google: its
physical network, its thousands of fiber miles,
and those many thousands of servers that, in
aggregate, add up to the mother of all
clouds.”
- Wired
Images by Connie
Zhou

Storage
Cloud Storage Cloud SQL Cloud
Datastore
Compute
Compute
App Engine Engine
App Services
BigQuery Cloud
Endpoints
Google Cloud Platform
Cloud DNS

Compute
Compute
App Engine Engine
Storage
Datastore
Power Computation
App Services
BigQuery Cloud
Endpoints
Cloud DNS

Storing What You Want, However You Want
Storage
Datastore
Compute
Compute
App Engine Engine
App Services
BigQuery Cloud
Endpoints
Cloud DNS

App Services
BigQuery Cloud
Endpoints
Building Robust, Intelligent Systems
Storage
Datastore
Compute
Compute
App Engine Engine
Cloud DNS

Google Compute Engine
• IaaS: VMs, Network, Storage
• Google DNA (speed, scale, reliable, secure)
• Fast Provisioning, Consistent Performance
• Enterprise Ready
• 24x7 Support
• 99.95% monthly SLA
• ISO 27001, SSAE-16 SOC 1,2,3
• Accessible Through
• Web @ https://cloud.google.com/console
• gcloud compute command-line utility
• REST API
• Partners (Commercial and FOSS)

Block Storage
Persistent Disks

Persistent Disk
10 TB
• SSD and "Standard"
• Data and root partitions
• Billed only by capacity (GB/month)
• Performance caps scale linearly with size
• Volume striping is automatic
• Differential snapshots
• Create new PDs based on snapshots

PD Use-cases
Root
Stateful root
volume
Root RW
Data
User managed data
volume
RO
Data
Instant distribution of
static content

Global Snapshot and Restore
vol1 vol2 vol3
GCS
vol1.
t2
vol2.
t2
vol3.
t3
t1 t1 t1
t1 t1 t1
t1 t1 t1
t1 t1
t1 t1
t2 t2
t2
t2
t3 t3 t3
● Point in time snapshot to Google
Cloud Storage (GCS)
● Differential snapshots
● GCS global replication!
● Restore from snapshot
anywhere in the world
us-central1-a
europe-west1-a

Machine Types Machine Type Virtual Cores
standard
• For workloads with balanced CPU and memory
highmem
• For workloads with higher memory requirements
highcpu
• For workloads with higher CPU requirements
Shared Core Instances
• For inexpensive prototyping and staging
workloads (g1-small, f1-micro)
Memory
(GB)
n1-standard-1 1 3.75
n1-standard-16 16 60.00
n1-highmem-2 2 13.00
n1-highmem-16 16 104.00
n1-highcpu-2 2 1.80
n1-highcpu-4 4 3.60
n1-highcpu-8 8 7.20
n1-highcpu-16 16 14.40

Images
+ Custom Kernel Support
(debian backports works great for docker)

Sub-hour Billing + Sustained Use Discount
1 minute granularity, 10 minute minimum
$
(30%)
Full billing cycle
Sustained Use Discount
● >25% use, price reduced
● Up to 30% reduction
● No commitment / contract

Enhanced Reliability
us-central1-a
scheduled maintenance event
Live Migration
• No downtime during scheduled
datacenter maintenance events
Automatic Restart
• Instances automatically
restarted if subjected to system
events such as hardware failure

Images by Connie
Zhou
Networking
• Google's Network
• Projects are isolated private networks
• TCP, UDP, ICMP only
• Multiple private network groups and firewalls
• Tags and address ranges
• Addresses
• public: static or ephemeral
• private: ephemeral with DNS
• Routes, gateways, VPNs, and IP Forwarding

Region: us-central1
Target Pool (tp-a)
Load-balancer
• Region based
• Session affinity, hashing options on tuples,
• Protocol, src ip:port, dst ip:port
• Consists of:
us-central1-a
www0
www1
tcp:80 ➔ tp-a
• Target pool: collection of instances
• Forwarding rules: protocol:port, pool
• HTTP health check: optional us-central1-b
www2
www3
Forwarding Rules
tcp:443 ➔ tp-a
Internet
http://googlecloudplatform.blogspot.com/2013/11/compute-engine-load-balancing-hits-1-million-requests-per-second.html

Noteworthy
• Regions
• Central United States, Europe, Asia (new!)
• Metadata
• Startup scripts
• SSH Keys
• Tags (instance/network)
• OAuth2 and Scopes
• Access other Google Cloud Platform services
• Ecosystem is growing
• Partners: RightScale, Scalr, New Relic, MongoLab, MapR, and many more...
• Open Source: Chef, Puppet, Salt, Ansible, Vagrant, Docker, CoreOS, fog, libcloud

Puppet DSL for Google Compute Engine
https://forge.puppetlabs.com/puppetlabs/gce_compute

How to Puppetize Google Cloud Platform - PuppetConf 2014

The Puppet Forge
a real world (meta) example

Abstraction Layers for Infrastructure as Code
Infrastructure Layer
Role & Profile Layer
Component Layer
Primitive Layer
puppetlabs-gce_compute
custom module
puppetlabs-postgresql
core puppet type/provider

Demo Architecture
1 3
2 4
us-central1-a
M
us-central1-b
allow tcp:80
Load-balancer
Target-pool
● [M]aster resides in GCE
● Uses gce_compute manifest to:
○ Spin up 4 VMs
○ Networking (firewall, LB)
● New nodes register with Master
● Install site.pp (apache+php+mysql)
● Custom index.php, connects to
Cloud SQL database
● Let's try it out!
Cloud SQL

Bring up the Compute Engine environment
puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
# puppet apply /etc/puppetlabs/puppet/manifests/gce_up.pp
# ...
# cat !$
1 $prefix = 'pc'
2 $zonea = 'us-central1-a'
3 $zoneb = 'us-central1-b'
4 $region = 'us-central1'
5
6 gce_auth { "graphite-demos": }
7

Bring up the Compute Engine environment (cont)
8 gce_firewall { "$prefix-firewall":
9 ensure => present,
10 description => 'Allow HTTP',
11 network => 'default',
12 allowed => 'tcp:80',
13 allowed_ip_sources => '0.0.0.0/0',
14 }
15
16 gce_httphealthcheck { "$prefix-http":
18 require => Gce_instance["$prefix-agent-1", "$prefix-agent-2"...],
19 description => 'basic http health check',
20 }
21 ...

46 gce_disk { [ "$prefix-agent-2", "$prefix-agent-4" ]:
48 size_gb => 10,
49 zone => "$zoneb",
50 source_image => 'debian-7',
51 }
52 ...

73 define create_instance_zoneb {
74 gce_instance { "${title}":
76 zone => "$zoneb",
77 disks => "${title},boot",
78 require => Gce_disk["${title}"],
79 puppet_service => present,
80 startupscript => 'pe-simplified-agent.sh',
81 metadata => {
82 'pe_role' => 'agent',
83 'pe_master' => "$hostname",
84 'pe_version' => '3.3.2',
85 }
86 }
87 }
88
89 create_instance_zoneb { ["$prefix-agent-2", "$prefix-agent-4"]: }

The Demo App
1 <html>
2 <title>PuppetConf 2014</title>
3 <body>
4 <h1>PuppetConf 2014: Puppetizing Google Cloud Platform</h1>
5 ...
6
7 <ul>This app is being served from host: <b><%= @ hostname %> [<%=
@fqdn %>, 7 <%= @ipaddress %>]</b></ul>
8 <p>Please enter a Nick and Message and watch it appear below:</p>
9 ...
10 <?php
11 $db = mysqli_connect("173.194.254.93", "puppetconf", " my_password",
"puppetconf");
12 ...
13 ?>

cloud.google.com
• Try out Google Cloud Platform with a $500 Credit
• Visit https://cloud.google.com/starterpack and use promo code puppet14-con
• Want more?
• Try the demo on your own at https://bit.ly/puppet-gce-demo
• or see http://googlecloudplatform.github.io/compute-video-demo-puppet
• We welcome your help and feedback to make Puppet + GCE even better!
• Find us at gce-discussion@googlegroups.com or
https://github.com/puppetlabs/puppetlabs-gce_compute

How to Puppetize Google Cloud Platform - PuppetConf 2014

More Related Content

How to Puppetize Google Cloud Platform - PuppetConf 2014