[Cloud OnAir] Talks by DevRel Vol.4 データ管理とデータベース 2020年8月27日放送

Cloud Onr
Cloud OnAir
Cloud OnAir
2020 年 8 月 27 日放送
【Talks by DevRel Vol. 4】
データ管理とデータベース

写真を配置後
角丸六角形くり抜きの図形を
被せてください
https://goo.gl/NcsiAz
Speaker
Cloud OnAir
Developer Advocate - Google Cloud
佐藤一憲

Speaker
Cloud OnAir
写真を配置後
上田哲広
@najeira
Google Developers Expert

写真を配置後
Speaker
Cloud OnAir
石村真吾
@sinmetal
Google Developers Expert

写真を配置後
Speaker
Cloud OnAir
Google Cloud
カスタマーエンジニア
岩成祐樹

Agenda
Cloud OnAir
1
3
2
4
データベース関連おさらい
Cloud SQL / Memorystore アップデート
Cloud Spanner アップデート
Cloud Firestore アップデート

Cloud OnAir
Cloud OnAir
データベース関連おさらい

Robert Kubis, Google Cloud, @hostirosti
DBS214
How to Choose the Right
Database For Your Workloads

01
Migrate
Lift and shift / optimize existing
workloads w/o relearning skills
03
Innovate & Transform
Extend and innovate on top of
databases build for the cloud
02
Modernize
Modernize and extend migrated
applications, rethink how you use and
simplify your databases in the cloud
Database journeys with Google Cloud
Benefits: Faster migration time, fewer
application rewrites, familiarity with
current stack, enable easier
modernization and innovation on our
platform
Benefits: Help reduce or eliminate
reliance on a single database vendor,
help reduce TCO of your database
deployments, adopt open-source and
cloud-native database technologies,
modernize database architecture
Benefits: Build powerful applications
and workloads that scale with your
needs, fully managed and integrated
with other parts of the platform

Homogeneous Migration Destinations
Compute Engine
Kubernetes Engine
Cloud SQL
Cloud Memorystore
Cloud Bigtable
Self Managed
Managed SQL
Managed Wide-Column
Managed Cache
Marketplace
Managed Partner Solutions
3rd party
Bare Metal Solution
Self Managed

Cloud SQL Features
● External Replication (MySQL)
● Backup / Restore backed by GCS
● SQL Server Replication
● SSD, high storage capacity → high IOPS
● Regional Disks
● Up to 96 vCPUs, 624GB Memory, 30TB storage
Database Migration Partners, Partner Services and Tools
How do we support your SQL migration
Google Cloud SQL
Fully-managed database service that
makes it easy to set-up and administer
your relational, MySQL, PostgreSQL,
& SQL Server databases in the cloud.
Bare Metal Solution
Enabling specialized workloads with
Google Cloud
Learn more: DBS102 - What’s new in Cloud SQL
DBS201 - Getting to Know Cloud SQL for SQL Server
DBS301 - High Availability and Disaster Recovery with Cloud SQL
https://cloud.google.com/solutions/database-migration

Purpose built
hardware
Flexible ways
to manage
Work with
Google Cloud
Low latency to
GCP Service
*Certified hardware link
Certified hardware* for
high-performance mission
critical workloads like
Oracle databases
Leverage existing DBAs or
bring your preferred SI
partners to manage
specialized workloads
Support and Billing from
Google Cloud for your bare
metal needs
Bare Metal Solution is built
in close proximity to
Google Data Center with
dedicated interconnect
Learn more: DBS213 - Bare Metal Solution: Bring All Your Enterprise Workloads to GCP
Bare Metal Solution with Google Cloud

Firestore
A fast, fully managed
serverless NoSQL document
database that simplifies
storing, syncing, and
querying data for your
mobile, web, and IoT apps -
in real time at global scale.
The power of our Cloud Native Databases
Bigtable
A fast, fully managed
wide-column / key-value
NoSQL database for use
cases where low latency
random data access,
scalability and reliability are
critical.
Cloud Spanner
A fully managed
enterprise-grade,
globally-distributed, and
strongly-consistent SQL
database service combining
the benefits of relational
database structure with
non-relational horizontal scale.
Learn more: DBS204 - Future-proof Your Business for Global Scale and Consistency with Cloud Spanner
DBS211 - Modernizing HBase workloads with Cloud Bigtable
DBS215 - Simplify Complex Application Development Using Cloud Firestore

Heterogeneous Modernization Destinations
Cloud Bigtable
Document Database Modernization
NoSQL Database Modernization
3rd party SQL
Cloud Spanner
Firestore
OLTP / SQL Database Modernization

How do we support your
database modernization?
Learn more: DBS208 - Switching from DynamoDB to Cloud Bigtable and Spanner
https://cloud.google.com/solutions/database-migration
Cloud Spanner
● JDBC / Hibernate
Support
● Spring Cloud Integration
● Local Emulator
● Dataflow Import
Templates
● Powershell Module
● Availability
● Scalability
Cloud Bigtable
● HBase API
● Local Emulator
● Terraform Integration
● Fast, high throughput
writes
● Availability
● Scalability
Firestore
● Used in front of traditional
databases
● Extending applications for
rich features like real-time
synchronization, offline
support
● Local Emulator
● Availability
● Scalability

How to choose your database
modernization destination?
Technical Considerations
● Tools and Integrations
● Data Model
● Consistency Requirements
● Ops and ease of use
● Availability
● Scalability
● Security
Business Considerations
● Reusing skills and knowledge
● SLAs
● Managed functionality and ops
● Security
● Compliance and Certification
● Pricing model
● Scalability
● Availability
Learn more: https://cloud.google.com/solutions/database-migration
Blog post on Medium: Things I Wished More Developers Knew About Databases

Why Cloud Spanner?
Practitioner perks:
● JDBC / Hibernate Support
● Terraform integration
● SQL Dialect Open Source
● Local Emulator
● API based management enabled
● Idiomatic client libraries
● cloudspannerecosystem.dev
Key features:
● Fully managed ACID relational database
● Built-in HA (SLA up to 99.999%)
● No planned downtime for anything
● Online elastic capacity scaling (qps, storage)
● Storage - pay for what you use
● Query Optimizer Version pinning
Key verticals:
● SaaS
● Retail
● Financial services
● Healthcare
● Gaming
Sample workloads:
● Inventory management
● Financial transactions
● Lots of real-time data
● Global meta-data
Learn more: DBS204 - Future-proof Your Business for Global Scale and Consistency with Cloud Spanner

Why Cloud Bigtable?
Practitioner perks:
● HBase Client compatible
● Terraform integration
● Key Visualizer for schema optimizations
● Local Emulator
● API based management
● Idiomatic client libraries
Key features:
● Fully managed wide-column NoSQL DB
● Row-level atomic operations
● Low latency random data access with linear scale
● Replication is eventually consistent across zones
and regions
● Zero touch failover for HA
Key verticals:
● AdTech, MarTech
● Financial Services
● Internet of Things (IoT)
● Media
● Retail
Key workloads:
● Real-time analytics
● Personalization, recommendations
● Monitoring (time-series)
● Geospatial datasets
● Graphs
Learn more: DBS211 - Modernizing HBase workloads with Cloud Bigtable

Why Firestore?
Practitioner perks:
● iOS, Android, Web clients
● Server Client Libraries
● Cloud Functions integration
● Expressive Querying
● Partition and Bulk Writer APIs
● Sync data across devices, on or offline
Key features:
● Fully-managed document NoSQL database
● Semi-structured / hierarchical data
● Online on-demand scaling
● Transactions on up to 500 documents
● Strong consistency across regions
● 1,000,000 concurrent real-time connections
Key verticals:
● Software & Technology
● Media
● Telecommunications
● Retail
● Mobile Games
Key workloads:
● User profiles
● User messaging
● Product catalogs
● Game state
● Offline/edge data
Learn more: DBS215 - Simplify Complex Application Development Using Cloud Firestore

Why managed OSS?
Practitioner perks:
● Use your existing skills and tools
● Wide variety of frameworks and connectors
● Idiomatic client libraries in multiple
programming languages
● API based management
● Integrated Monitoring
Key features:
● Fully managed open source databases
● Seamless updates (OS and DB system)
● Integrated with the rest of our platform
● Managed Backup/Restore
● Flexible resource allocation
● HA using Regional Persistent Disks
● Point in time recovery (MySQL, PostgreSQL)
Key verticals:
● SaaS
● Retail
● Financial services
● Healthcare
● Telco
Sample workloads:
● Inventory management
● Financial transactions
● Cache
● Search indexing
● SaaS applications on top of OSS databases

Why BigQuery?
Practitioner perks:
● Reuse existing skills - SQL
● User-defined functions
● Federated Queries
● NL query interface
● Multi-cloud integration
Key features:
● Serverless fully managed data warehouse
● Partitioned columnar storage
● Streaming inserts
● Supports MERGEs, UPDATEs, DELETEs
● Declarative machine learning in SQL
Key verticals:
● Retail
● Advertising
● IoT
● Gaming
● Financial Services
Key workloads:
● Interactive reporting on large data
● Data Science and advanced analyses
● Big Data / ML processing using SQL
Learn more: DA200 - What's New in BigQuery, Google Cloud's Modern Data Warehouse

Metadata management
service
In-memory Non-relational / NoSQL Relational Data warehouse Partners Databases On GCE (IaaS)
Microsoft SQL Server
Oracle
SAP HANA
IBM Db2
MySQL
PostgreSQL
Non-relational
Cloud
Memorystore
Managed Redis
Query with
microsecond
latency
Ideal For:
Real-time
analytics,
caching
Firestore
Serverless,
scalable
document
Store
Quickly query
on any
attributes
Ideal For:
client side
mobile and
web apps,
gaming
leaderboards,
user presence
at global scale
Cloud
Bigtable
Low latency,
scalable wide
column store
High throughput,
low latency
reads and writes,
endless scale
Ideal For:
Real-time
access, large
analytical
workloads and
building
low-latency
applications
Cloud
SQL
Managed
MySQL,
PostgreSQL,
& SQL Server,
Bare Metal
Solution
ACID
transactions,
Referential
integrity
Ideal For:
WordPress,
backends,
game states,
CRM tools
Cloud
Spanner
Scalable
relational
database
Horizontal
scale, strong
consistency,
ACID, SQL,
endless scale
Ideal For:
scale insurance
and
mission-critical
availability
BigQuery
Enterprise
data
Warehouse
Serverless
managed
data
warehouse
Ideal For:
real-time
analytics,
advanced
and
predictive
analytics,
enterprises
Built and managed by Google Built and managed by
Partners
(as a service or Anthos)
Managed by you and/or Partners
Data Catalog
Data management portfolio

Document Key-value In-memory
Wide
Column
Graph Time-series Search Relational
Data
warehouse
Cloud
Firestore
Cloud
Bigtable
Cloud
Memorystore
Cloud
Bigtable
Cloud
Bigtable
Cloud Spanner
Cloud SQL
BigQuery
Amazon
DocumentDB
Amazon
DynamoDB
Amazon
ElastiCache
Amazon
Keyspaces
Amazon
Neptune
Amazon
Timestream
Amazon
Elasticsearch
Amazon Aurora
Amazon RDS
Amazon
Redshift
Azure
CosmosDB
Azure
CosmosDB
Azure Cache for
Redis
Azure
CosmosDB
Azure
CosmosDB
Azure
CosmosDB
None
Azure SQL
Database
SQL
Datawarehouse
Coverage by Database Category

Cloud OnAir
Cloud OnAir
Cloud SQL アップデート

What's New With Cloud SQL
Brett Hesterberg, Google Cloud
DBS102

A fully-managed database service that
makes it easy to set-up and administer
your relational, MySQL, PostgreSQL,
& SQL Server databases in the cloud
What is Google
Cloud SQL?

BigQuery to Cloud SQL federation
Easily analyze data
● Enables BigQuery to query data
residing in Cloud SQL in
real-time, without copying or
moving data
● After the initial one-time set up,
you can write a query with the
SQL function
EXTERNAL_QUERY()
● Last month Cloud SQL
federation brought in 21PB of
data to BigQuery

Committed Use Discounts
Ideal for workloads with predictable resources needs
Fully transferable between MySQL, PostgreSQL, and
SQL Server
Fully transferable between machine sizes
One Year
Commitment
Discount: 25%
Three Year
Commitment
Discount: 52%

Disaster Recovery:
PostgreSQL & MySQL
Cross-region replication
● Guard against a region failure
● Google Cloud VPC is global
● Secure: Private networking
● Easy to use: Create a replica in
any region in the world with no
additional networking
configuration

Point-in-time
Recovery:
PostgreSQL

New:
Maintenance
Notification &
Rescheduling
Prepare for upcoming maintenance
● Receive notifications one week in
advance of incoming maintenance
activities
Reschedule maintenance
● Delay maintenance up to one
week or apply it immediately

Cloud OnAir
Cloud OnAir
Memorystore 最新情��

Gopal Ashok, Google Cloud
DBS210
Optimally Deploy
an Application Cache
with Memorystore

Cloud
Memorystore
Redis and
Memcached
(Beta) managed
by Google
Open source redis and
memcached managed
by Google
Highly available
and reliable
Replication
Automatic failover
Availability SLA
Scalable and
secure
Automated scaling
Private IP deployment
Network security
Granular access
control (IAM)

Memorystore for Redis
Max Capacity 300 GB
Tiers Basic, Standard
Availability SLA 99.9%
Zonal Replication ✓
Automatic Failover ✓
Data Backup ✓
OSS protocol compliant ✓
Versions 3.2, 4.0, 5.0
What’s New ● Version 5.0
● Version upgrade
● Shared VPC Support
● VPC-Service Control

Memorystore for Memcached (Beta)
Max Capacity 5 TB
Horizontal Scaling Up to 20 nodes
Node size Up to 256 GB/ 32 cores
Availability SLA 99.9%
OSS protocol compliant ✓
Versions 1.5.16
Discovery Endpoint ✓

Cloud OnAir
Cloud OnAir
Cloud Spanner アップデート

Cloud OnAir
● DBS204-JP
○ Cloud Spanner で進化し続けるビジネスを支えるシステムを構築
● DBS303
○ Lessons Learned from Evaluating Cloud Spanner at Uber Scale
● DBS205
○ Modernize Your Gaming Applications for Scale and Player Engagement
Spanner

Cloud Spanner とは、リレーショナルデータ
ベースの構造の利点と、非リレーショナルの
スケーラビリティを組み合わせた、
新しいデータベース
● スキーマ、ACID トランザクション、
SQL をサポート
● スケールアウト / インが任意に実施可能
● フルマネージドなデータベース
● 非常に高い可用性を提供
○ SLA 99.99 % 〜 99.999 % を提供
○ メンテナンスダウンタイム無し
Cloud Spanner の特徴
性能
スケーラビリティ
SQL
運用管理

Cloud Spanner の構成
ゾーン a ゾーン b ゾーン c
Cloud Spanner
インスタンス
2 ノード
東京リージョン (asia-northeast1)
分散ストレージ
(Colossus)
(Colossus)
(Colossus)
Cloud Spanner は、コンピュートとストレージが分散されており、デフォルトで
すべてのゾーンにサーバーが配置される
コンピュート
ストレージ
Server 1 Server 2 Server 1 Server 2 Server 1 Server 2

Proprietary + Confidential
● テーブルは主キーでソートされる
● テーブルはキー範囲で分割される
● Split は負荷とサイズに応じて
分割される
Split とは？
Server 1 が
管理
Server 2 が
管理

インターリーブ ( 親子関係 )
● データベースには複数のテーブルを定義できる
● データを物理的にコロケーションしたい場合
● 最大で 7 階層まで
Interleave とは？
CREATE TABLE UserItems (
UserId STRING(36) NOT NULL,
ItemId STRING(36) NOT NULL,
Num INT64 NOT NULL,
) PRIMARY KEY (UserId, ItemId),
INTERLEAVE IN PARENT Users
ON DELETE CASCADE

インターリーブした場合のデータのロケーション
インターリーブした場合インターリーブしない場合

胡淑鳳 (Sophia Hu)
データマネジメントスペシャリスト
DBS204 -
Cloud Spanner で進化し続ける
ビジネスを支えるシステムを構築

2019 にリリースした Cloud Spanner 新機能
エンタープライズ機能
VPC-SC
アクセスの透明性
単一ノード SLA (99.99%)
デベロッパーの生産性
OSS JDBC
Spring Data と Hibernate ORM
Cloud Spanner エコシステム
SQL サポート
コストベースのインデックス選択
バッチ DML
WITH 句

2020 にリリースされた Cloud Spanner の新機能
エンタープライズ機能
マネージドバックアップ - 復元
マルチリージョン構成 (Asia 1 & Eur 5)
Change Watcher ソリューション
デベロッパーの生産性
エミュレータ
C ++ クライアントライブラリ
トランザクション統計
Harbor Bridge
SQL サポート
クエリオプティマイザーのバージョン
外部キー

マネージドバックアップ / 復元 - GA
オンデマンドマネージドバックアップと復元
● DB 単位でバックアップ取得可能、オペレーター /アプ
リエラーからの障害復旧に対応
● バックアップはスタート時点に対して一貫性を持って
います、1 年まで保存可能
● 別のインスタンスに対して高速にリストア可能
● データベースの別ドメインとして、
バックアップと復元に対する独立な権限管理
● API, Console, gCloud 経由で利用可能

ローカルエミュレータ - GA
デベロッパーが、オフラインのエミュレータによってローカル開発およびテストが可能になった
● メリット
○ アプリケーション開発コストの削減
○ デベロッパーの生産性の向上
● 特徴
○ オープンソース
○ Spanner のクライアントライブラリをすべて利用可能
● 対応可能なテスト
○ インスタンス、データベース、SQL、DML、DDL、
○ セッション、トランザクションレベルのテスト
■ マルチバージョン同時実行制御 (MVCC) の読み取り
● GCP 上のインスタンスとの違い
○ エラーメッセージ、実行プラン、外部キー、
パフォーマンス、ロックの挙動など

クエリオプティマイザーのバージョン - GA
ユーザーは特定のバージョンの Query Optimizer を使い、クエリを実行できるようになった
Cloud Spanner のオプティマイザーは、効率的なクエリプランを
作成するのに確立されたヒューリスティックとコストベースの
最適化を組み合わせて使用
3 つのオプションクエリオプティマイザーバージョンを固定可能
● DB レベルでの設定
● クライアントライブラリのグローバル変数を変更
● SQL クエリの実行中にヒントを使用

Nandakumar Gopalakrishnan, Uber
Ramachandran Iyer, Uber
DBS303
Lessons Learned from
Evaluating Cloud Spanner at
Uber Scale

Request Patterns
● Avoid Hotspots - Completely random primary key is
best for avoid hotspots & for simple CRUD like
operations.
● Interleaved Tables - For relationship, better to use
interleaved-tables
● Scans - For scan operations, better to have the
following
○ Avoid scanning the same rows over
○ Use timestamp & shardId combination

● Reduce R/W transaction time (first operation to
commit time)
○ Increase in R/W transaction time = increase in
lock contention
● Retry on Aborts
○ Identify what to abort and what not to
● For R/O transactions, use snapshot reads
Transactions

Transaction Execution
r
w
c
r1
w1
c
w2
r2
r1, r2
w1
c
w2
r1, r2
w1, w2, c
Basic txn
(3 round trips)
Multi entity txn
(5 round trips)
Multi entity txn +
TransactionGraph
(4 round trips)
Multi entity txn +
TransactionGraph +
TransactionCoalescer
(2 round trips)

gRPC
● Context deadlines
● Flow control
● Compression
● Interceptors to the rescue
Client-Side Tuning
Sessions
● Read-Write vs Read-only sessions
● Session Management APIs are
expensive
● Pool size based on workload

● Scales horizontally
○ CPU utilization
● Ease of management and Provisioning
○ Multi-region support
● Network
○ Routing/Bandwidth is bigger bottleneck than Spanner
● Database Splits and Compactions
Spanner-side Tuning

Sophia Hu, Google Cloud
Takayasu Oyama, COLOPL, Inc.
DBS205
Modernize Your Gaming
Applications for Scale
and Player Engagement

Split 1where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
id: 1
id: 2
…
…
Latency increases for Split 1
Split 2
id: xxx
id: xxx
…
Split 3
id: xxx
id: xxx
…
Cloud Spanner
HOTSPOT
Handling burst of access to shared data

where id = 1 and copyNo = 1
…
…
Split 2
id: xxx
copyNo:2 id: 1
…
Split 3
id: xxx
copyNo:3 id: 1
…
Split 1
id: xxx
copyNo:1 id: 1
Cloud Spanner
(possible solution)

Shared Data
(Master)
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
where id = 1
Replica Set
(actual solution)

Game Bots
Simulated Queries
Latency (99th percentile)
Storage Split Split SplitSplit
Can't split fast enough!
Cloud Spanner
Handling launch day traffic

Game Bots
Storage Split SplitSplit SplitSplit
Simulated Queries
Triggers "load based split"
Cloud Spanner
Handling launch day traffic

Cloud OnAir
Cloud OnAir
Cloud Firestore アップデート

Cloud OnAir
● DBS215
○ Simplify Complex Application Development Using Cloud
Firestore
Cloud Firestore

Minh Nguyen, Google Cloud
Arash Parsa, Google Cloud
Marta Kosarchyn, Khan Academy
DBS215
Simplify Complex
Application Development
Using Cloud Firestore

Performance best practices
def next_id() {
next_id += 1
return next_id
}
def handle_request(doc) {
db
.collection(‘Foo’)
.document(next_id())
.set(doc)
}
key-space
time
~1000s of RPS
shards
single shard

Performance best practices
def handle_request(doc) {
db
.collection(‘Foo’)
.add(doc)
}

Avoid heavy sequential patterns
on indexes
doc = {
...
‘time_inserted’: ‘1593041978’
...
}

Cloud OnAir
Cloud OnAir
まとめ

Cloud OnAir
● DBS214: How to Choose the Right Database For Your Workloads
● DBS102: What's New With Cloud SQL
● DBS210: Optimally Deploy an Application Cache with Memorystore
● DBS204-JP: Cloud Spanner で進化し続けるビジネスを支えるシステムを構築
● DBS303: Lessons Learned from Evaluating Cloud Spanner at Uber Scale
● DBS205: Modernize Your Gaming Applications for Scale and Player Engagement
● DBS215: Simplify Complex Application Development Using Cloud Firestore
取り上げたセッション

[Cloud OnAir] Talks by DevRel Vol.4 データ管理とデータ ベース 2020年8月27日 放送

More Related Content