A presentation at Agile 2013 in in Nashville, TN, USA by Karthik Gaekwad
How DevOps changed everything Karthik Gaekwad @iteration1 Wednesday, August 7, 13 #AgileDevOps
Howdy! • I’m Karthik Gaekwad • Senior Web Engineer • Mentor Graphics Embedded • From Austin, TX • We have great BBQ, chill people and brutal summers! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
My Background • National Instruments (NI) • Ecommerce Team (Agile) • R&D Cloud Team (DevOps) • Mentor Graphics Embedded • Cloud Team (DevOps) • Organizer: CloudAustin, DevOpsDays Austin, TheAgileAdmin @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Audience Survey @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Let’s not waste time! The most important DevOps things I’ve learned over the last 4 years @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
First, we have to talk about agile… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
There was a waterfall Requirements Design Implement, Test Deploy Maintain @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Roles Requirements Product Manager, Business Analyst Design Business Analyst, Web Designer, Architect Implement Programmer Analyst, Web Developer Deploy Programmer Analyst, System Administrator Maintain System Administrator @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Waterfall.. S @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Communication “But I told you this already” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Accountability “That is not my job!” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Time to deliver “That is not a priority” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
**About the time I started… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
New Product: Partslist Collaborate and tag parts to build large systems @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Transitioned to SCRUM 2 Week Iterations Targeted Feature sets Demo at end of iteration @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
The Product Team > Traditional Organization @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Results Working together, building things that were wanted @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Results Between Business and IT @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Results Better products being delivered, at a faster pace. Methodology expanded to all other web teams… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Results I love my job! #ForReal @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Agile: Retrospective • Scrum brought team together. • Building products on time, which were actually needed. • Superior collaboration between business and IT. • Organization shift -> Scrum @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Ops conversations “Agile is cool! but it’s hard for ops to be agile because it takes months to buy and provision servers.” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
@iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Ops conversations Deployer Dude “All day everyday” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
But in the community… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
But in the community… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Go build us cloud products “We’re making cloud now” products now” Sincerely, Management @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Cloud is “We’re making cloud AWESOME!” products now” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Cloud is “We’reOSSM” making cloud -Dave Nielson products now” -NIST @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Cloud is AWESOME OSSM” On-demand “We’re making cloud Self-service products now” Scalable Measurable @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Give me API or give me death” @littleidea @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Agile DevOps “We’re making cloud Cloud Team” products now” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps “A cultural and professional movement” Adam Jacobs @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps “System administrators participating in an agile development process alongside developers and using many of the same agile techniques for their systems work.” The Agile Admin Blog @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Get rid of the devs; get rid of the ops; let’s just find people who can solve problems” Alison Chaiken Embedded Linux, Thought Leader @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Pillars of DevOps CULTURE AUTOMATION MEASUREMENT By: John Willis SHARING Damon Edwards @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps 101 Ideas and Stories Free!! No charge!! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
“Agile DevOps “We’re making cloud Cloud Team” products now” Buzzword friendly! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
We had a goal: “Build a cloud authentication prototype” Wednesday, August 7, 13
R E U LT U C Our 1st sprint(s) sucked! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Devs: “REST Services!” But we suffered from “Wall of confusionitis” Ops: “System Stability!” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Not seeing eye to eye #Frustrating #Angry @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Really, this is “Platform Architecture”, but I’m too lazy to redo the image @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Devs and ops understand good architecture. #GoodPlaceToStart @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Think platform first, then app later. #PaaS #REST @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N A U T O M A T 1st win: Architecture • Everyone understood architecture • Helped everyone speak same language • Reduced friction • Called it “The System Model” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N M A T T O A U 1st win: Architecture • “Model Driven Automation” • Championed by ops • Everyone needs to understand the model • Figure out how to deploy the model @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N M A T T O A U • 1st win: Architecture Programmable Infrastructure Environment (PIE) • Based on version controlled architecture diagram • • • • Provisioned cloud boxes Ran deploys Set standard naming for boxes On demand deploy for apps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N M A T T O A U Outcomes • Ops learns Java! • Ops learns version control! • Developers get comfortable with production environments! • …Now we’re speaking many common languages! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
A U T O M A T IO N Are y’all crazy??? No Chef? No Puppet? @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
A U T O M A T IO N Everybody be cool • We believe the cleanest abstraction layer is starting with the entire system you’re trying to model • In 2009, chef and puppet were younger • We were scared of Ruby • We needed windows cloud support • Plus dawg, cloud was a wild wild world then! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N M A T T O A U Please automate!! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IO N M A T T O A U Do I pick Chef or Puppet?? “whatever makes you happy” @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IN G R SH A For chef/puppet, you’ll need to code. #NotJust4Devs #NotScary @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IN G R SH A Stuck? Just ask your DevOps dev guy! #NotJust4Devs @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IN G R SH A Don’t hide. Share the knowledge @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IN G R SH A Wednesday, August 7, 13 App configs
IN G R SH A App configs Easy Change Management Let your automation control these kind of files Wednesday, August 7, 13
PS D EV O Tools Philosophy • Needed to work for cloud • Facilitate collaboration • Charts and graphs • Everyone wants to use it @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
E R SU EA M Tools Philosophy • Monitoring = Cloudkick • Monitor servers • Monitor code (custom synthetic checks) We use #datadog now @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
E R SU EA M Tools Philosophy Teach your devs how to monitor their code, and they will love the tool Example: #NewRelic @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
M EA SU R E Tools Philosophy If it measures business metrics, management will love you too! #statsd #dogstatsd HINT: Management pays you. Make them happy @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
E R SU EA M Tools Philosophy WASTE OF TIME: SSH’ing into a box to read log files @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
M EA SU R E Tools Philosophy Get a logging solution. Plethora of choices! #Logstash #Graylog #Splunk #Sumologic Which one? CloudAustin July Meetup @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
IN G R SH A Tools Philosophy You’ll need more tools Find them or write them! You’re devops now! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
E R SU EA M Tools Philosophy DevOps victory: “People” use tools that solve their core problems @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps 201 MOAR Ideas and Stories For ops who can code and dev not scared of linux @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Is different for every team. What works for your team, works for your team @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C This worked for us… #AlwaysBeInclusive @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Stuff Breaks Like…all the time @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C #DontDoThis @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT U C Be Cool. Fix the issue at hand If you panic, everyone else does too. @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
C U LT U R E Post Mortem Do a Post mortem @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT C U Post Mortem It’s okay to admit mistakes You’re not Batman… @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT C U Post Mortem Why x5 Add more tests Make sure it doesn’t happen again @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
R E U LT C U RELEASE OFTEN Release with 4 changes is less risky than Release with 40,000 changes @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O Wednesday, August 7, 13 Testing
D EV O PS Test Everything Test the code Test the infrastructure It’s all the same now #infrastructureAsCode @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O Free Time? Write more tests! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O CONTINUOUSLY @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O CONTINUOUSLY @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O Manual Changes To Production @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
PS D EV O Manual Changes Don’t do this. Do this: 1. Make manual changes in dev 2. Add to your chef/puppet script 3. Release to test/prod @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps 301 MOAR MOAR Tips For when your team grows @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Security in the loop Add security audits as a part of your sprint. We do an audit periodically @Gauntlt @RuggedDevOps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Security in the loop Hire an infosec pro to your devops team @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Security in the loop No budget? Try a cross functional working group @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Expanding Teams? • Rotate times for meetings (status/planning) • Scrum standups were recorded and dropbox(ed) • “OK to call” philosophy • Campfire chat (saves history) • Ownership of new features @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Expanding Teams? • No secret: Face to face is always better • Prioritize travels for visits to teammates @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Everyone asking for DevOps? The embedded DevOps model works* @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Everyone asking for DevOps? Make sure your team culture is strong enough so you don’t lose person permanently :) @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Start Small… Don’t say “WE’RE 100% DEVOPS TODAY” Wednesday, August 7, 13
DevOps 401 MOAR MOAR MOAR Ideas For next time. I don’t have any. Tweet #agiledevops for your best ideas! @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
STILL STUCK? Just ask! We’re devops, and this is real (even though we can’t define it) Twitter: #devopsama http://devopsweekly.com/ Go to a devopsdays https://groups.google.com/forum/#!forum/devops @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
DevOps: Retrospective @iteration1 Wednesday, August 7, 13 • DevOps and CAMS bring team together. • Building stable products on time, which were actually needed. • Superior collaboration between devs and ops • Organization shift -> DevOps Agile 2013 #AgileDevOps
DevOps: Retrospective • DevOps and CAMS bring team together. • Superior collaboration between business and IT. Building stable products on time, which were actually needed. • Superior collaboration between devs and ops Organization shift -> Scrum • Organization shift -> DevOps • Scrum brought team together. • Building products on time, which were actually needed. • • @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
Thank you! Let’s work together, and solve the problems that our business wants us to! #DevOps @iteration1 Wednesday, August 7, 13 Agile 2013 #AgileDevOps
The most important DevOps things I’ve learned over the last 4 years. I presented this at Agile 2013 in Nashville, TN. This talk is the talk and story referenced in Gene Kim’s Devops Handbook (https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002)
#agile #devops #automation #culture #distributedTeams #measurement #sharing #bestPractices
for free. You
can too.