Configuration Management Isn't Everything

Editor's Notes

1. This talk started as a conversation between a fellow consultant and I. Often we go into shops and they think DevOps (or Chef, whatever) is something you can just order from Amazon.com or NewEgg and suddenly your business will be accelerated. Maybe this is Stockholm Syndrome from years of enterprise software sales reps telling them “just buy our software, we will solve XYZ for you”
2.   - Revolutionizing IT in an org is gonna take a lot more than just buying/implementing a CM product.   - It’s amazing how many leaders don’t get this. But they have the purchasing authority.
3. Janky scripts - “There’s nothing more permanent in IT than a temporary solution”
4. Speed Bureaucracy Poor working conditions - Usually this is at the SA level, demoralized because they feel like they’re just line cooks, and business is always breathing down their neck even though they’re engaging in heroics to keep the systems alive
5. “Automate all the things” is not a useful goal for either executives or sysadmins. You could replace “automate all the things” with “cloudify all the things” or “devops all the things” or any other buzzword, it still doesn’t make for a sensible objective. Automating for the sake of automating is pointless. Automation is the medicine for your headache; have you actually solved why you have a headache in the first place?
6. How many of you have read the Phoenix Project? How many of you could relate to it? I’m going to give away the ending a little bit here, but you should still read it, if for nothing else to know that you aren’t alone in your experiences in IT. Able to keep up with speed of biz, responsive etc. Owning your IT and being in control of it will let you leapfrog over competitors that don’t have the same competencies. Doesn’t matter whether you are Tesco or Deutsche Bank. Those of us in IT often say “we’d like to improve communications between the biz & IT” because we just get unrealistic orders dumped on our lap. Well, here’s how you do it -- when IT is part of the biz, it’s a lot harder for the business to treat you as a cost centre or short order cook. If you’re a pharmaceutical company, for example, you’d never ask your scientists to run a clinical trial & ship a drug in a month. You have some idea how long it takes and have clearer expectations because that’s your business.
7. Timelines and Effort: * “We want to move everything to the cloud in the next two quarters” -- well think about how long it took to build that stuff in the first place. And how do you know you can move everything? What about your EOL AIX servers? Your AS/400s? * Unprepared for the amount of change required to fully automate. Scope of change: I say “one foot on the dock and one foot on the boat” -- not 100% committed to change. And if you ride a bike too slowly it will fall over. For example, the client agrees that automation is great, but they later freak out when they realize they have to give up direct control over things. I’ve seen customers flip out about not making manual changes on the server or revoking ssh or sudo access to admins -- all the things are done through Chef. Well, are you buying into the consistency & reproducibility guarantees of Chef or not? Existing processes that they want to shoehorn CM into. “How can we implement Chef within the confines of our existing ITIL process?” Security team freaks out and doesn't want anything running as root. Client doesn't even own their own hardware, therefore making changes is very painful when they have to depend on a third party. Have you had those conversations with the MSP? These are financial and contractual implications that customers don’t think about but stand in the way of a successful implementation. And just to touch on “magic”, sometimes that’s our fault as salespeople -- or in your organization, if you’re “selling” CM up to management. Next slide:
8. I said I wasn’t going to swear in this presentation... sorry. Paraphrasing a wise person who may or may not be in this room...
9. Jez Humble: http://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/ Some customer engagements go badly because staff are pulled off to go do “real work” -- guess what, the CM system *is* the real work from now on.
10. Own the changes - what happens if I come in, “eat shoot and leave?” Next time you need to make a change you’ll be lost. “Hire goats and give them rope” -- Michael D.
11. How many of you are cat people? This story will be familiar to those of you who own cats. Once upon a time in a certain village in India there lived a guru.  Every evening the guru would sit on his seat and deliver a lecture to the public. It so happened that the guru had a cat, and just at the time of giving the lecture the cat would create a big disturbance. Being greatly annoyed by the cat, the guru decided to tie the cat to a tree before starting his lecture. So doing, the guru then delivered the lecture without disturbance. It worked so well that the guru regularly tied the cat to the tree before beginning his discourse.  After some years the guru died. His disciples carried on the guru’s program. They also continued tying the cat to the tree.  When the cat died, they bought another cat and thus the ritual of tying a cat to a tree continued generation after generation.  In the fifth generation that followed the guru, one of the renowned followers wrote an elaborate treatise on the spiritual significance of tying a cat to a tree before beginning one’s studies of the scriptures.  It's tradition!
12. If you analyze why ITIL, CABs, etc. and other layers of bureaucracy exist, it’s often to “guarantee” safety in IT systems because we had no other way of doing so. So managers often want
13. Big bang with process change is sure to get people’s backs up.
14. Don’t pick the biggest, most gnarliest project up frontMake change incrementally -- no “big bang” implementation. It is a lifestyle changeChoose a route that will minimize likelihood of failure -- analogize to high-speed rail projects in America Tell the story of high speed rail in North America and why it’s so bad compared to Europe & Japan
15. MongoDB or document-oriented databases -- “schemaless” doesn’t mean “don’t plan the schema ahead of time”
16. Model is not only a diagram/inventory of what you have and roughly how it was built, but what data structures, cookbooks, etc. you are going to use and have for your CM tool. Do forensics on your running infra, if necessary -- that’s why we often suggest to customers that we do an “infrastructure assessment” before embarking on a CM project. Developing a model on the fly is harder than having a baseline to start, even if the baseline changes down the road. Workflow’s importance: what I learned from working in the media, where reporters had the most screwed-up workflows and it inhibited them from producing great content as a result.
17. You can’t do automation (or any IT projects) by force -- dead-drop dates don’t work. You may just end up launching garbage and end up fixing them incrementally in the worst possible way: publicly. Also, Gene Kranz *never* said this IRL. That was made up by the Apollo 13 writers… what he did say was “When bad things happened, we just calmly laid out all the options, and failure was not one of them. We never panicked, and we never gave up on finding a solution.”
18. Again, why you don’t choose the riskiest, most gnarly project to start with. Have buffer room for failure & experimentation. Not failure of the entire project, but room to experiment & not build up too much tech debt.
19. Easy experimentation -- branching and merging ops can be done completely offline, and you can rewrite history & rebase before publishing Collaboration & collaboration were popularized by GitHub and you can either do that, or BitBucket, or Gitlab, or a myriad of others If Mercurial, Stash (Git server variant), Perforce, etc. let you do these things, then use them if they make you comfortable
20. Many companies implement peer-to-peer chat systemsLync, Skype, GChat, etc.What’s lacking is group chatHipChat, Campfire, IRCWhy group chat? (list examples here, link to Mark Imbriaco’s GitHub ChatOps talk) - Need to make the case for why
21. Publish your builds to an artifact server* Enforce policies on the artifact server (immutable objects, who can publish builds, etc.)* Mirror external dependencies there (RubyGems, Java libraries, etc.)Objective: every time you install a system or even “mvn compile”, you get the same thing* Examples: Artifactory, Sonatype Nexus
22. WSUS - Windows Server Update Services
23. Maybe here is where we say “just add water” doesn’t work Tell them, not gonna sell you on DevOps culture or anything, that’s not what this conference is about “Every presentation must have a pie chart, so I have one” -- Sean Carolan
24. “Reinforce culture with technology” - Adam
25. CM is not a magic bullet for all your IT problems. But many companies think that it is.