Turbocharge your automated tests with ci
- 2. WHY AM I QUALIFIED TO BE
UP HERE?
• President of OpenSource
Connections
• Contributor to CruiseControl
and Continuum CI projects
• Member of Apache Software
Foundation
• Presenter at conferences
(OSCON, ApacheCON, jTDS,
ExpoQA, STPcon 2009!)
- 7. AGENDA
CI in a Flash
Value Proposition of CI
Challenges in using CI
Scaling CI for LARGE suites of tests
Wrap Up
7
- 9. DEFINITION
a fully automated and reproducible build, including testing, that
runs many times a day - Martin Fowler
http://martinfowler.com/articles/continuousIntegration.html
- 10. MARTIN FOWLER
• Coinedthe term
Continuous Integration
• Chief
Scientist for
ThoughtWorks
• Speaksand Blogs on
software:
• http://martinfowler.com/
- 12. MAPPING
diagram credit to Brian Di Croce @ http://blog.briandicroce.com/2008/03/17/setting-up-a-continuous-integration-
environment-preface/
- 13. FEEDBACK MECHANISM
From: cruisecontrol@opensourceconnections.com
Subject: [CruiseControl] agenda_direcciones build 370 fixed
Date: October 7, 2008 2:22:50 PM EDT
To: epugh@opensourceconnections.com
The build has been fixed.
CHANGES
-------
New revision 370 detected
Revision 370 committed by epugh on 2008-10-07 18:22:03
mysql esta mejor que sqlite
M /branches/ERIC_EXPO_08_DEMO/agenda_direcciones/config/database.yml
See http://cruise.opensourceconnections.com:3333/builds/agenda_direcciones/370 for details.
- 15. ACTIVITY
Turn to the person next to you, and see how many
non-obvious things you have in common.
- 16. LIFE WITH OUT CI
• Unstable code, integration is difficult
• Many build errors reported
• Only one person on the team can
build the entire project
• Getting “test ready” state is hard
• Feedback cycle is long
Every day is a struggle to be productive
- 17. LIFE WITH CI
• The build process is simple
and repeatable
• Elimination of Human Error
• Bugs stay fixed!
• Feedback cycle is very rapid
Every day you know you can produce better software!
- 19. RISK REDUCTION
• What is in the build?
• What has changed between
builds?
• How do I verify
functionality?
- 22. REDUCE REPETITIVE
PROCESSES
• DRY: Don’t Repeat Yourself
• Script Everything!
• “ant deploy”
- 23. MAKE PAINFUL THINGS
PAINLESS!
photo credit to Paul Duvall @ http://www.integratebutton.com/
- 24. PRODUCE DEPLOYABLE
SOFTWARE
• It’s what we are here to do!
• Always in a “Demo Ready” state
• No last minute administrivia tasks to get the product
ready
• Installer is built
• Documentation is written
• Changelog is written
- 37. FEAR OF COMMITMENT
• Developers and Testers need to
commit changes frequently!
• Incremental small changes make
finding integration issues simpler
drawing credit to Tom Armitage @ http://infovore.org/archives/2007/03/14/the-continuous-integration-barometer/
- 40. • CI needs a champion who acts as the ambassador
to the leaders of the organization.
• Thought leaders in the organization who can help
encourage team to accept the process changes that
CI brings
•A very successful test case
•A greenfield project if possible
- 41. CI ON A DOLLAR A DAY
"From now on, our code in revision control will always build
successfully and pass its tests.". - James Shore
• Build Computer
• Ridiculous Toy
• Annoying Bell
• Automated Build
• Group Agreement
http://jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html
- 42. credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/
enterprise_ci_cultural_maturity.html
- 44. TESTABILITY
• Modularity? • Perform code complexity
analysis to see if some parts
• Coupling to underlying are more amenable to
platform? testing.
• Requiresexternal services
to be up and available for
testing?
- 47. CUP OF COFFEE METRIC
What happens in the time it takes to get a cup of coffee can
tell you a lot about the quality of your continuous integration
system. - Paul Julius
• Builds shouldn’t take any longer!
• Gettinga cup of coffee or drink
of water breaks up your day
• Givesyou a chance to review
your work
• Requires a moduler fast build
http://pauljulius.com/blog/2009/09/14/cup-of-coffee-metric-for-continuous-integration/
- 48. PAUL JULIUS
• co-founder of CITCON
• Original
developer of
CruiseControl
• Speaksand Blogs on CI
extensively
• http://pauljulius.com/blog/
- 49. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&
- 50. GETTING TOO FOCUSED ON
GREEN BAR
Glare of the Green Bar in CI can
obscure issues with your product
James Bach
- 52. INVEST IN CI SERVER
X
• The CI system must be as well maintained as
the Source Control system.
• The CI system performs builds very quickly.
✓
• Who has responsibility for CI? It’s very
important to have a “throat to choke”.
• No false alarms. If there are false alarms then
the team loses confidence in the CI system.
•
- 56. !"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
F)&/)5+.15'/#" D-&&A+.15'/#)* .)&GH.)5I'1)+ L-"(H,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#'1+
;3)+!3IJ L54E4#'43"
24"#&A+ D-&&A+.#$3*$5*'K)*+
.#$3*$5*'K)* 0)/&4A"
- 57. !"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&H,$")*+ .'&4)*+:)/45#'3C F'"#45'1$&+ 95)3*'3C L5)*'1#'I)
:)/45#" :)/45#" 24*)&'3C
L-%&'"(+ 654""H.'&4
>'"'%'&'#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"
- 59. MULTI STAGE CI
• Reduce the amount of
integration required
• Teamsintegrate as modules
are marked as ready.
• Maven2 was built with this in
mind!
Damon Poole @ http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html
- 60. BREAK UP TESTS
• “preflight” or “smoke” tests
• longer integration tests
• periodic full site generation and report generation
- 62. PARALLELISM
• Ifyour suite of
tests take 24
hours to run, can
you spin it up on
24 machines and
take an hour?
• Testing a web
application with
Selenium?
Selenium Grid!
- 65. PREDICTIVE TESTING
• Some commercial tools support predicting which tests will fail
based on previous behavior, and running those first!
- 66. DON’T DO IT ALL AT ONCE
• Justlike when embarking on a new exercise program, don’t
just go full tilt, work up to it!
• Start
with the tests that are easily put under CI first, then
move up the food chain.
- 67. SLIDING “WINDOW”
• For a test suite that takes a
week to run, can you take a
sliding chunk, plus some
random tests?
• Ideas
from DNA
sequencing!
• http://en.wikipedia.org/
wiki/DNA_sequencing
- 68. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'I)+ RSST+
9)"#+?-#4E$#'43 9)"#"+:-3+Q'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64I)5$C)
,-'&*
?-#4E$#)*+ !@#)3"'I)+
D-31#'43$&+9)"#" ?-#4E$#)*+
D-31#'43+9)"#"
:'"BH,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"
- 69. LOTS OF
ADVANCED
TECHNIQUES
!
credit to Christine Mitterbauer @ Microdoc
- 71. !P!2!89.+;D+!89!:L:7.!+67
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&
!"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
F)&/)5+.15'/#" D-&&A+.15'/#)* .)&GH.)5I'1)+ L-"(H,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#'1+
;3)+!3IJ L54E4#'43"
24"#&A+ D-&&A+.#$3*$5*'K)*+
.#$3*$5*'K)* 0)/&4A"
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'I)+ RSST+
9)"#+?-#4E$#'43 9)"#"+:-3+Q'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64I)5$C)
,-'&*
?-#4E$#)*+ !@#)3"'I)+
D-31#'43$&+9)"#" ?-#4E$#)*+
D-31#'43+9)"#"
:'"BH,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"
!"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&H,$")*+ .'&4)*+:)/45#'3C F'"#45'1$&+ 95)3*'3C L5)*'1#'I)
:)/45#" :)/45#" 24*)&'3C
L-%&'"(+ 654""H.'&4
>'"'%'&'#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"
'!()*+,"$-!).$/ !"#$%&"!'
!"#$%&'()*&)+(&,%-$(.(/0$&12&3"-+(/-4&3)5$6&78/-$.".$(
!""#$%%&&&'()"!*++#,-'.-/%!"/+%,01-2,.01%&!*"03#(#0,1
- 72. Don’t forget the people aspect!
credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/
enterprise_ci_cultural_maturity.html
- 77. CI IN A BOX
• Amazon EC2 Hosted instance
• http://www.ciinabox.com/
- 80. CITCON
• Open Spaces Style
• Twice a year, US and International
• Portland, Oregon, April 2010
• http://www.citconf.com/
- 83. THE BOOK
Continuous Integration:
Improving Software Quality
- 84. THE CI GUYS
http://ci-guys.com/
Free online consulting
- 85. THANK YOU!
• twitter: dep4b
• speakerrate: http://www.speakerrate.com/epugh/
• email: epugh@opensourceconnections.com
85