The Challenges & Pitfalls of Database Continuous Delivery

#
Uri Margalit
Director of Products

#
Uri is a product executive, who brings over 15 years of
experience and a proven track record in taking new ideas to the
market. With a passion for understanding and translating market
requirements into working products and solutions, Uri is able to
effectively lead Product teams in the enterprise software,
systems management. Uri earned a BA in Sciences from the
Open University.
In his spare time, Uri plays chess and enjoys gardening.
.

#
• Doing better with less
• Reacting quickly to market needs
• Getting ahead of competition
• Just can’t wait 6 months for that next release…
– Agile Development
– Process Automation
– DevOps

#
Fail
Pass
Image from Wikipedia

#
• Rapid changes
– Reacting quickly to market needs
– Getting ahead of competition
• Fewer changes backed out
• Better collaboration
• Fewer defects
• Ultimately better service
• Happy customers
• Profitability

#
• Team and process
• Version everything
• Automate your tests
• Fix it, properly (no out of process changes!)
• Automate your deployments
• Create the deployment pipeline

#
• The database holds your essential information
• Changes can impact the entire system
• Need to be synchronized with other changes
• Often overlooked

#
• There is more to a database than SQL scripts
– Schema structure
– Code
– Content and meta-content
– Internal dependencies
– …
• Ensure that changes are not made without
authorization
• Ensure no out-of-process changes

#
• Old adage but true
• The database is often neglected and therefore can
become the weakest link
• Essential from a compliance point of view
• Should be the strongest link

#
• Silos exist…
• Don’t always communicate effectively
• Need to share knowledge
• Need to follow same procedures & best practices

“
“
#
We recently had a disaster - the script in the version
control was not updated and when executed in production,
ran the wrong revision. That cost tens of thousands of $
(an out-of-process update to QA that was not properly tracked)
DBA @ Algo Trading company
We had multiple releases to production every day. That is
one release a week with multiple follow up fixes, and yet more
fixes
(code overrides, partial versions, wrong versions – all pushed to production)
CTO @ Credit Card company

“
#
We had an incident where a trigger was not correctly
implemented during a code release. a trigger from a previous
build was used instead which was only detected on Tuesday
morning on the first business day after our code release. this
was a customer-facing application and made our team look,
and feel, bad about the release.
we realized that we needed to bring more discipline and rigor
to our database changes.
(manual process are hard to repeat over and over without errors)
Sr. DBA @ Large USA Bank

#
• Manual script based version control process
• Deployment tools unaware of version control
• No red-flags…

#
Version Control Process
(file based)
Development Process
Check-Out
Script
Modify Script
Get updated
Script from
DB
Check-In
Script
Compile
Script
in DB
Debug Script
in DB
?
?
A
?
?
A’

#
• Challenges…
– Code-overrides
– Working on the wrong revisions
– Scripts do not always find their way to the version
control solution
– Out of process updates go unnoticed
– Hard to locate outdated update scripts

#
Int QA Stage Prod
11..1234567 111...147 11..11.1
X
1.1.1
Database Deploy Script
Re-Base (due to defects)
Environment
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process
Change
X
X
X
X
X
?
X

#
• Scripts, unless super sophisticated:
– Unaware of changes made in the target environment
– Time passed from their coding to the time they are run
– Potentially overriding production hot-fixes & work done
in parallel by another team
• Content changes are very hard to manage
– Metadata & lookup content does not practically fit into
the VC
– In most cases they are simply not managed

#
• Integrated Version Control process
– Leverage proven version control best practices
• Forcing check in & out for changes
• Labels
• etc..
– No code-overrides
– Always working with the correct revision
– All changes are documented

#
• Integrated Version Control process
– Always know who did what, when, why and from where
– No out-of-process changes
– Supporting structure, code and content
• No time spent on manual coding of the change
scripts

#
• Correlate each database change with a change
request
– Task ID
– Work Item
– Trouble Ticket
– CR
– etc…

#
• Partial deployments (a feature, a collection of
bugs, etc…)
• Scope changes easily synced between code and
database

#
For deployment automation we need:
Automatic, repeatable, safe process for deployment
automation
Using tools make sense …

#
Int QA Stage Prod
11..1234567 111...147 1.1.1 11..11.1
Database Deploy Script
Re-Base (due to defects)
Environment
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
Out of Process
Change
1.7
*
1.7

#
Test cases using compare & sync tools:
An index exists in source (QA) but not in target
(Production)
What should we do? Add the index or not?

#
• An index exists in Target (Production) but not in
source (QA)
• What should we do? Drop the index or not?

#
Safe to automate?
No. Requires manual inspection…

#
Simple, right?
NO! we are going to BREAK production without even
knowing…

#
• A compare & sync tool:
– Is unaware of any changes that occurred before the
time it ran
– Has no knowledge of changes that took place at the
target environment
– Does not leverage version control for more information
– Unable to deal with conflicts & merges between different
teams

#
• A compare & sync tool:
– … Requires manual inspection
– Requires detailed knowledge regarding each change as
part of the process
So… no automation… as automating problems
into production is a major risk!!!

#
We need to leverage version control into
deployment decisions…

#
Simple Compare & Sync Baseline aware Analysis
Source vs.
Target
Action
= No Action
≠ ?
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have
all of the
information
With baselines aware
analysis the unknown is
now known

Development Baseline
Previous Label /
Production Golden Copy
# 45
Production
No index in baseline =>
we should protect the NEW index on production!!!
(Protect Target)

Development Baseline
Previous Label /
Production Golden Copy
# 46
Production
BUT… If we had the index in the baseline =>
we should take it down from production…
(Deploy Change)

#
Deployment Automation
And Merge???

#
• Database Safe Automation:
– Leverages version control
– Has a flexible scope
– Can be run as a batch process
– Integrates to ALM (labels, CRs, Continuous Integration
& Delivery)
– Deals with conflicts & merges to match code agility
Can raise red flags to stop the line…
if requires human intervention

#
• Database Enforced Change Management solution
– Database version control
– Enforce best practices
– Plugs into the ALM (change request, tickets & work
items)
– Database merge & change impact analysis
– Know who can do what, where, when & why

#
• DevOps Solution for databases
– Baseline aware deployment automation, rollback &
recovery
– Reduce database deployment issues
– Plugs into release management & Continuous Delivery

#
• Database Enforced Version Control
– Enforced Check Out/In
– Labels
– Rollback/Undo
– Audit trail reports
• Database Impact Analysis
– Utilizes version control repository information
– Baseline aware analysis

#
• Database deployment automation
– API for automation
– Baselines
– Conflict resolution
– Customized business logic

#
Uri Margalit
urim@DBmaestro.com
@UriMargalit

The Challenges & Pitfalls of Database Continuous Delivery

More Related Content

The Challenges & Pitfalls of Database Continuous Delivery